I had an idea about a high-efficiency self ballasted MH lamp. The lamp would feature a halogen lamp as the ballast to increase efficacy and lifetime, a pulse start MH burner and Yttrium vanadate coating to convert some of the UV light to visible light. An ignitor utilizing today's technology would be built inside the lamp.
I think efficacy in exccess of 30 lm/W could be achievable, with very good CRI.

The uses of such lamp would be
-replacement for incandescent or SBMV, with higher light output and better color.
-new installations where low cost, simple wiring, good color rendering, instant light after turn on, good power factor or low inrush current is desired.

Well, just an idea...

I think there would be an issue with arc stability: The time, when the mains voltage is lower then the arc voltage is quite long, what would lead to long "dark" interval. During this interval ionisation decay (as it is not supported by the electrical field), what would increase the reignition overshoot time (time, what it take for the available OCV to rebuild the ionization after the mains voltage exceed the arc voltage).

The inductance of the choke ballast play very important role: It's inertia keep the arc current long past the mains voltage zero-cross, what mean when the current drop to zero, the mains voltage (opposite polarity) is already quite high (about 250V). So when the current stop flowing, there would be no di/dt anymore (steady 0 has zero di/dt), so no voltage across the coil. So the lamp would immediately see 250V across electrodes. So here both the time is short, as well as voltage high for the reignition, so the lamp have no problem to reignite reliably (in fact it ignite way sooner, then the lamp voltage reach the mains level)

To overcome this, you would have to provide some means of "plasma keeping" during the "dark time". Simplest to do this is to separate the arc from the mains by diode rectifier, in parallel generate higher voltage (by multiplier) and feed this via high resistor to the lamp. Then if the main current drop, this small current would keep the plasma alive, so at the time the mains voltage rise again, the thick arc would establish without any significant overshoot. This method is so efficient, it allow the arc voltage to way be more then 50% of he mains rms voltage even with the resistive ballast (generally stated limit for choke ballast is, then the arc voltage should not exceed 50% of the mains rms voltage).
But the drawback is, you operate the lamp with only one polarity, so you need DC arctube.
And if you have it, use standard incandescent filament as a ballast, packed inside common outer with special DC arctube, you get exactly this ...

Medved, thanks for the detailed reply, now I understand it better

@bluelights: Actually, there is a self ballasted MH lamp exist: The US Philips MasterColor CDM Integrated PAR38 Protected, which available in Europe as Philips MASTERColour CDM-i.
This is a PAR38 ceramic MH lamp that have an integrated electronic ballast and is of 25W.

@But it is an electronic ballast, where i would fear all the pain associated with CFL's...

Medved: But unlike CFLs, ballast overheating shouldn't be an issue in a self ballasted PAR38 CMH lamps, as MH lamps, produces far less convection heat then a CFL and most of the heat from MH lamps are IR radiation (MH are point light source unlike CFLs).
So the ballast should be run cooler in a self ballasted PAR38 CMH lamp.
This is also the reason why electronic drivers in LED lamps, haven't the problems of ballast overheating in CFLs.
The only heating problem in LED lamps, are the LEDs themselves, and it is easily solved by heatsinking.

It does produce far less convection heat and on top of this PAR is expected to be operated uncovered, but simplest MH ballast is more complex then simplest CFL ballast and it should fit into the same, or even smaller area. This would ask for minimum size components, so no significant design margin (like in e.g. OSRAM DULUX LongLife) is feasible.
Moreover it would be highly sensitive to mains overvoltage spikes: Unlike higher power ballasts, this does not have high value electrolytic capacitor (it won't fit there), what serve as quite good overvoltage clamp (it does not swallow all ISO test-pulses, but most switching disturbances it does, if the capacitance is 47uF and above) in higher power electronic devices...

Something like this ?

http://lamptech.co.uk/Spec Sheets/GE First Light.htm

Medved, it does actually work to make a tungsten-halogen self-ballasted metal halide lamp.  I have made some before and they do not extinguish due to the long dark period and reignition peaks do not become excessively high (at least for the higher wattage arc tubes and 230V mains).  With a neon-argon filled arc tube of self-starting design like in Philips' old HPI-BUS then you can even have a totally electronics-free solution!  With HPS however it does not work, due to the problems you mention.  Especially if this is with a neon-filled arc tube, then during run-up the lamp will extinguish due to increasing reignition peaks.  In HPS the thermal losses to the wall are greater than mercury or MH and the importance of the longer 'off' period with a resistive rather than inductive ballast becomes apparent.  Philips has 3 or 4 patents detailing various phase-shift devices in the lamp cap to compensate for that problem and appears to have developed the idea quite thoroughly, but even so has never marketed a self-ballasted HPS.

But on 230V (still assume 100V arctube) the losses on the ballast part would be way more then 50%, what would cut the efficacy to half. Moreover the target application was household incandescent replacement, so package of about 1500..2000lm total, so MH tube wattage about 35..50W

The GE DC arctube concept with the ionisation keeper allow the arctube to drop more then 70V on 120V mains, so I would guess more then 140V on 230V with the same concept. Of course, it is not "100% electronic free", but the question is, if few diodes, capacitors and some resistors could be still called "electronic" (if there are no active components), but I know what do you mean...
I think doing such concept today does not make much sense, when there available quite reliable active components allowing to design e.g. the high frequency ballast, with a bit of design effort it may be electrolytic-capacitor free (a bit oversized electrodes to allow higher current crest factor and the ballast does not need then the problematic tank capacitor on the DC bus), so still without any thermally sensitive and short living components...

By the way James: Do you know, what problems to expect when driving modern, low wattage (~35W, even ceramic) MH tubes by DC? They seems to be quite short, so the diffusion/convection mixing should prevent separation of the gas fill by the electrolysis in the gas. Let's assume the strict vertical position (to avoid liquid fill being subjected to the current) is acceptable, as well as the negative potential of the arctube (versus the surrounding)...

Wouldn't it be possible to make a self-ballasted MH lamp with a resistive ballast (like tungsten-halogen) and a built-in glowbottle starter to start the lamp?

There are HPS lamps with built-in glowbottle starter.

Wouldn't it be possible to make a self-ballasted MH lamp with a resistive ballast (like tungsten-halogen) and a built-in glowbottle starter to start the lamp?

There are HPS lamps with built-in glowbottle starter.

But these need the inductor in series, without the inductance in series with the lamp the starter is useless: It is the ballast inductance, what generate the necessary high voltage ignition pulse for the lamp when starter contacts open. With the resistive ballast there would be no HV pulse, so the starter would be flashing widely, but the lamp would not ignite...

So if it have to work with simple resistor (e.g. an incandescent filament) in series, the lamp should be able to in the first place ignite directly on the mains (like MV), without any HV aid (so no glowbottle like OSRAM's SON-I, nor thermal starter like Tesla used on HPS retrofits for MV).
Second condition to work on resistive ballast is the reignition spikes should be small in order to be able to maintain the arc, so the HPS would not work (according to James, I didn't try it, but such statement correspond with the fact, then when HPS is only worn out, o operate a bit hotter then designed, it is not able to maintain the arc anymore, so cycle even on series choke, what is able to handle reignition spikes nearly up to the mains peak voltage).
And this requirement for limited reignition spike height apply even for operation with quite long (1..2ms) gap without any current, as unlike the inductor, the resistor is not able to reverse the voltage on the load as quickly (with resistor the voltage follow the mains)

I had an idea about a high-efficiency self ballasted MH lamp. The lamp would feature a halogen lamp as the ballast to increase efficacy and lifetime, a pulse start MH burner and Yttrium vanadate coating to convert some of the UV light to visible light. An ignitor utilizing today's technology would be built inside the lamp.
I think efficacy in exccess of 30 lm/W could be achievable, with very good CRI.

The uses of such lamp would be
-replacement for incandescent or SBMV, with higher light output and better color.
-new installations where low cost, simple wiring, good color rendering, instant light after turn on, good power factor or low inrush current is desired.

Well, just an idea...

Your idea would work as with any idea but it's about whether the companies are willing to invest in a product that will fail efficiency standards so they would rather use more expensive electronics that are efficient but don't last long and charge a high price for it because they used state of the art ballasting to achieve the same results as using a filament lamp in series with the arctube. Would be a cool bulb to have and use too bad the world is run by Philips Lighting and Osram who only think of profits.

Your idea would work as with any idea but it's about whether the companies are willing to invest in a product that will fail efficiency standards so they would rather use more expensive electronics that are efficient but don't last long and charge a high price for it because they used state of the art ballasting to achieve the same results as using a filament lamp in series with the arctube. Would be a cool bulb to have and use too bad the world is run by Philips Lighting and Osram who only think of profits.

All businesses around are here to make profit at first place. In the former "Eastern block" run for a half of a century an "experiment" trying it in the "social responsible" way and I think the result is still visible even after more then 20 years since the "experiment" ended...
So I would not change that system...

The reason for incandescent and MV demise is their retail cost being pushed unfairly low, so it didn't allow any profit.
Over last 50years their nominal cost remain the same, but the inflation was more then factor of 5...
Moreover the lighting product portfolio really exploded, spreading the demand, so making the volumes of each particular product too low. But they are afraid to stop any product line alone, as it may move their customers elsewhere. So they are trying to "synchronize" so, there would be no maker of such dropped product, so customers would shift towards some of the the remaining one.
And as trusts are forbidden, they used the "green" wave and do that via lobbying for regulation laws.