How is the HID/LPS ballast so inefficient? My theory is that when a HPS bulb has 100w with 9500lm (95lm/1watt), the ballast will increase the wattage up to 185w to 205w, and keep the 9500lm the same, which will make the lumen efficiency 45lm/1watt to 50lm/1watt. And that's what makes the ballast so inefficient. Is my theory incorrect and what's the real reason why the HID/LPS ballast is inefficient.

Gear losses are complicated 😬, I’ve tested a 35 watt SOX on low loss gear with a total circuit draw of 56 watts, and the same lamp on autoleak transformer with a total of 65 watts!
No discharge lamp is actually as efficient as they are on paper 🤔

How is the HID/LPS ballast so inefficient? My theory is that when a HPS bulb has 100w with 9500lm (95lm/1watt), the ballast will increase the wattage up to 185w to 205w, and keep the 9500lm the same, which will make the lumen efficiency 45lm/1watt to 50lm/1watt. And that's what makes the ballast so inefficient. Is my theory incorrect and what's the real reason why the HID/LPS ballast is inefficient.

Where did you get that 100 and 200W figures?

I guess now is the time to refresh some basic circuit theory! First chapters of the famous Horowitz & Hill Art of Electronics book have some good explanations, for example! Do a search, there are pdf versions of that book floating around the net.

What are you using to measure it?

With any gas discharge lighting, measurements are made more complicated by introducing odd wave forms and bad power factors.

A proper watt meter should compensate for this, but you never know quite sure what those plug-in watt meter things are actually doing internally.
Old mechanical kWh meters definitely should take into account wave shape, but - and CMIIW - should ignore phase shifts between current and voltage. This actually worried people who bought LED lighting with capacitive droppers, because those things have an atrocious power factor and newer digital kWh meters can actually take the power factor into account.

It might very well be that you have no power factor correction capacitor on your ballast and you end up with a power factor of 0,5 or so. You can compensate with putting a capacitor parallel to mains of the correct value. Typically the right value is printed on the ballast itself or otherwise provided by the manufacturer. Once you get power factor to 1, measuring gets a bit easier again.

The "old mechanical energy meters" actually do exttract the real power correctly, provided they are properly calibrated (the phase shift compensation). Their problems were rather high internal power consumption (few W in a single phase voltage coil) and inability to correctly measure/track low power levels (20W and below), regardless of the power factor/THD, it is just a thing of mechanical friction.

But phase shift won't cause wrong reading, neither high harmonic content. The disc torque is just plain product of eddy current field and the load current field at any instant, so once the voltage coil field is calibrated well (the one responsible to form the eddy currents in the disc), just plain physics makes sure the true power is registered well.

HID ballasts losses use to be between 5..20%, depends on the exact system.
The simplest, cheapest and most efficent are series chokes, when the arc voltage is close to the half of the mains (for a stable arc the mains should not be lower than twice the arc, but the system needs room for lamp aging).
That is, why the lower power HPS in the US/Canada market and pretty much all HIDs in the "230V" area are designed this way, with 55V arc for 120V HPS, 70..90V for "230V" area HPS, about 100..140V for "230V" MV and MH.

Then for lamps with higher arc voltage in the 120V area you need some form of step up transformer and these have vay more windings, so exhibit higher losses. And because of the more windings, they are heavier and more expensive. But the higher open circuit voltage allows the lamps to be more efficient, so these are the main choice for all but the low wattage HPS in the 120V world.

The more complex ballasts allow also designs with the ability to compensate for wider mains voltage fluctuation, which makes the installation mainly with longer wiring simpler, but it costs some extra losses in the ballast (generally the better the regulation, the higher the losses).

The LPS very difficult lamps to power in an efficient way. Unlikethe high pressure lamps, their anode column voltage drop is high from the first ionization, so if you add the high drop of the cold cathodes just after ignition, the ballast needs to beable to deliver quite significant power into a discharge with voltage drop many times higher than the normal operation (after electrode warmup). That means very high OCV requirement. And that means even on 230V, the ballast needs to be more complicated, so (unless some HF electronic gets involved) more lossy. So much the losses use to be in the 50..100% of the rated lamp power (so the ballast efficiency as low as 70..50%). In the latest years, electronic ignitors able to provide the higher voltage feed for the cold cathode stage operation made possible the LPS to suffice with a simple series choke ballast, then the efficiency became closer to the 70..80% (the LPS atill have rather low arc voltage, so the ballast losses remain higher).
Of course completely HF electronic ballasts are able to boost the voltage for the cold cathode operation using resonance, so retain high efficiency for normal operation, but these came too late, when the LPS technology was already on the way out from the market, so there was no motivation anymkre to invest into their adoption.

@Eleco_SR304 did not reply anything on how he measures it, but 2X figure hints he may be measuring apparent power as V*A product without any correction to power factor applied.

the reason is : P_losses=I² R _ballast+P_magnetic

P_magnetic~U²_ballast

For the rest get yourself a book about basic electronics...