I was wondering why mercury free HPS lamps were so much less common than “standard” lamps. If mercury free HPS production expanded, it might have discouraged some LED conversions, as they are actually more efficient than standard HPS and don’t contain anything highly toxic. Still, I think LED would have continued to take over unfortunately at some point, but I’m still interested.

In the US and Canada, they were uncommon. In Europe they were common.

The "mercury free HPS" could be called a scam to some extend. The problem is, they perform way worse in many ways, mainly are less efficient. This lower efficacy means, to power that extra power you need, with the energy sources used for the electricity generation, you release way more mercury (from its trace content in the fuels, mainly coal) i to the environment than over the life cycle of the more efficient "tradditional" ones. Assume a reasonable waste management of the spent lamps (with extraction and reuse of the amalgam material - not that difficult to clean the mercury up for reuse in new lamps), of course, not them just being dumped into the landfill.

The "mercury free HPS" could be called a scam to some extend. The problem is, they perform way worse in many ways, mainly are less efficient.

Calling Hg-free HPS lamps a "scam" is a gross exaggeration stemming from ignorance. They were in fact more efficient than standard HPS lamps (140 lm/W vs. 123 lm/W, 400 W lamps) and lasted quite longer too (38 vs 28 kh to 50 % failure, 400 W lamps). However, it is correct that mercury free lamps did have lower performances than those of mercury-filled "plus" or "super" HPS lamps, but the difference was not that significant (141.5 vs. 140 lm/W, 400 W lamps). That's for state-of-the-art "Plus APIA" lamps, but compared to conventional "Plus PIA" Hg-filled lamps, the Hg-free variants outperformed them (140 vs 137 lm/W, 400 W lamps). The same goes for the service life, the Hg-free variant lasted longer than the conventional "Plus PIA" Hg-filled lamps (38 vs 36 kh, 400 W lamps) but not as long as the state-of-the-art "Plus APIA" (45 kh). All data given here are from Philips, 2014 (since they ended the production of Hg-free lamps in 2015).

Also, on the aspect of light output and power consumption, your reasoning does not apply to the reality of how Hg-free lamps were used, lighting installations were not designed around mercury-free HPS lamps. Those lamps were in fact used to retrofit existing installations, so their usage did not result in any increase of the power consumption, only the resulting light level changed (increased or decreased depending on what lamp type was used before). As far as recycling is concerned, in an ideal world we should indeed collect all spent lamps to prevent the release of their toxic materials, but we are not living in an ideal world and a significant fraction of lamps still end up in landfills.

In the US and Canada, they were uncommon. In Europe they were common.

Your evidence for this sweeping statement, please..?

Most mercury free HPS lamps, were made by European manufacturers. American manufacturers almost didn't made these lamps except Osram Sylvania.

That's no evidence, it's only a assumption. If we go by your way of reasoning, then lamps are common in China but not in the rest of the world because most of them are made in China?

Exactly Max, the world thrives on the import and export of manufactured goods.
Unfortunately the days of making goods for the home market is not the normal practice any more.