LED chip size is about 1mm, the lens uses to be about 10mm away from the chip.
With 5cm long arc in the HPS the equivalent would be 0.5m radius of the reflector with the HPS...
But it does not have to be that far for the angle accuracy, the typical 10cm radius of the HPS lantern is usually sufficient and that correspond to 5mm die size (the LED's are never that big, it would be too much heat and light concentration and scrap rate due to chip defect density) for the same 10mm radius lens.

The LED lenses do count on the light distribution of the LED's they are designed for. It is not always just a flat surface, as on many LED's there is a small lens directly on the chip, making the light intensity more constant along the angle.
 But indeed, it may cause compatibility problem, when someone uses older lens designed for older LED beads with the small ball lens on top, with some new LED with optically flat surface or vice versa. The resulting characteristic them may lead to light distribution problems (glare,...)


And of course, none of the optic is 100% ideal, but the additional losses would be comparable for both LED and HPS.

The LED chips used in the "lens panel" LED lanterns are on the order of 1W..3W each. I am pretty sure the chips are multiple die, coated with a common square of phosphor. That is much more than 1mmx1mm.. so not a point source compared with the ~20mm lens

The HPS arc tube is 6mm x 50mm, the reflector size is about 1ft x 1.5ft. That is. in the X direction (along road, controlling over illumination loss) precision is excellent (6/300), in the Y direction (across road, controlling spill) it is still fair (50/450), and possibly better than the LED if chip size is >2.5mm in 20mm lens



The LED package (i expect...) is designed to handle the heat transfer from the LED to the metal panel behind it, sufficient for the power of the LED. Then the limiting factor is the heat transfer from the panel further out, that will only affect minimum possible distance between the LEDs (to keep W/Area manageable in the lantern design). but is no more a matter of the design of the LED itself, so i dont see how it limit the power of one LED chip ?



Everything can be made wrong, i am not about that. Assuming the LEDs and lens panel are made to match each other correctly, i can imagine the limits of the lens panel design getting its efficiency down, maybe well to the same level as HPS optics, allthough from different factors than the HPS

Then if the optical efficiency is comparable, what is the deal with assuming that LED is 1.0 optically efficient and everything else not ?

The LED chips used in the "lens panel" LED lanterns are on the order of 1W..3W each. I am pretty sure the chips are multiple die, coated with a common square of phosphor. That is much more than 1mmx1mm.. so not a point source compared with the ~20mm lens

Up to 5W for the general lighting applications there is only one single die per a component, only for special purpose (projectors, long throw narrow beam lights,...) even higher power per single chip. But of course higher power means larger chip and that means there is higher chance it will be affected by some manufacturing defect (those are randomly spread across the wafer, the only known parameter is their statistical density; if the defect happens to be on an active area of any component on the wafer, that component just does not work).

The LED package (i expect...) is designed to handle the heat transfer from the LED to the metal panel behind it, sufficient for the power of the LED. Then the limiting factor is the heat transfer from the panel further out, that will only affect minimum possible distance between the LEDs (to keep W/Area manageable in the lantern design). but is no more a matter of the design of the LED itself, so i dont see how it limit the power of one LED chip ?

The package is able to deal that heat, but then the heat has to continue further till it reaches the ambient. And there are the problems: To transfer it to the ambient, you need certain area of the heatsink surface. And for that to work, you have to spread the heat evenly across that surface.
Now assume you have 50x50cm heatsink.
Then if you have only single 5x5mm square for 100W dissipation, you have to first distribute the heat from the small square 25cm away to reach the end of the heatsink (still ignored the diagonal distance being larger). This adds up quite significant heat resistance, or needs very thick mass or other means (heat pipe,...) to distribute the heat better.
If you split that power into 25 just 1x1mm pieces, each dissipating 4W and you distribute these evenly across the heatsink, what you get is each piece dissipating 4W having it's own equivalent 10x10cm piece of the heatsink. That means you have to spread just the 4W and just 4cm away (again ignoring the diagonal distance, as in the previous case). For that even rather thin piece of aluminum will ensure very low thermal difference, so the overall thermal resistance would be way better and still without any thick distribution metal or heat pipes or so, so really very easily you get excellent thermal performance. And that is the main aspect, which makes lower power, but higher count approach cheaper, even when it means dealing with more components.

Of course, this splitting has it's limit, for e.g. a projector you can not utilize anything else than the light from a single source, therefore the high power single chips.
For the streetlight the best compromise seems to be somewhere between 2 till 5W chips. But the thing is, if you take an assembly designed for 2W chips 10 years ago and use there todays LED's (e.g. you have proved this design is cheap and reliable), it will thermally work the same as 10 years ago, you get the same thermal performance. But as the 2W LED used to emit 60lm, but today's 2W chip emits 240lm, your assembly becomes 4x brighter and that may happen to be the difference between "below" and "above" the glare limit.

Then if the optical efficiency is comparable, what is the deal with assuming that LED is 1.0 optically efficient and everything else not ?

What I mean is:
Both HID, as well as LED do have extra light losses (absorbtion, spill) on top of the principal optical working, I wouldn't expect these losses to be any different. I would expect these losses to be somewhere in the range of 10% or so for both.
What differs is, if we assume ideal optic with both light source styles, the HPS will emit 30% of it's light outside of the target illumination area (for certain geometry of the area vs pole height), while the LED won't (you are able to control all the light).
So if we take 90% efficiency from the optical nonidealities, plus the 70% efficiency of an ideal HPS optic vs 100% efficiency of an ideal LED optic, what we get is 63% total efficiency for HPS, vs 90% total efficiency for LED's.

I see a skipped step in the straight away assumption that LED optics have 0 "principal" losses



Example of losses that i dont think are avoidable :

View of lantern from below (1 of the lenses)

Y
^
|  ....
| ( # )
|
|
+-------> X

<-----> along road

To make the 2 main beams, forming the X sides of the lens - the ( ) is straightforward

But the chip emits in Y angled directions too, ie towards the .... edge of the lens. The lens must be complete, so it have a wall on that side too. at whatever angle the wall is, it is parallel to X, so it cant redirect the light in X. The beam will stay normal to X, so the only choice is between throwing the light down or outside of the road in front/behind the lantern...

The typical LED lens format for road lighting. You can see it it is fully enclosing the LED chip in all directions the LED radiates, hence shaping the complete light.
So yes, the lens has to reach till around the sides of the LED as well, but as you see, it is not that difficult.

This type of lenses is frequently made in the form of a single panel, featuring an array of such lenses, where it is designed to have one LED chip behind each of the lens mounted on a flat PCB placed underneath the lens panel...

This exact LED lens size can thermally tolerate 3W LED's, but can not be used above 100lm per lens, as it would lead to the too high surface brightness, so glare.

The places where i see possibility for same sort of losses are :

In Y direction :


The light emitted at wide Y angle from the chip is hitting the side wall of the lens (the "inwards V" wall). That wall must direct it down in Y direction, or else it will end up outside of the road

The refractive index of the lens is only about 1.5. This limits how much down can the lens direct the light that comes in at so big Y angle. The lens may not be able to change the angle that much efficiently (the higher the angle of beam hitting the lens from inside is, the more light will be reflected back from the lens-air interface and less go through, so for efficient working the lens cannot be put at very high angle either)



In X direction :


The light emitted into the "inwards V" is most of the light of the chip

Only small part of this light is needed down there, that passes through the rounded corner of the V

The rest of the light have to be allready thrown to the sides as wide as possible. But with the light hitting allmost normal to the lens, it can only so much redirect this light to the side. So there is possible over illumination spread over area (i estimate by eye) at about 45 deg, ven when the light directly under the lantern is ok

The LED in the picture you linked to is shown from about 30 deg X angle. The area of chip visible through the lens in OFF position, gives estimate to the amount of light that will go in ON in the same direction. The chip is indeed "shrinked" by the lens ie. less light goes in this direction compared to without the lens, but it is not particularly small either

The first picture shows the use of a lantern arrangement designed to illuminate the road+sidewalk on places without the sidewalk. Then of course the light thrown on the nonexistent sidewalk is getting lost. There could be good reason for that: If most of the installations require the sidewalk to be illuminated, having extra lantern for some short stretch without without wouldn't make much sense.

The second shows what happens, when the LED is not inserted deep enough inside of the lens. Many lenses (e.g. the one I put link before) are designed for the older style LED packages, where the LED itself is about 2mm above the board level, so inserted inside of the lens structure. Now when you put modern LED (with barely 0.5mm above the board level) on the board there, the maximum beam will close together to smaller angle, so instead of covering the distance they form an over illuminated bands closer to the lantern.

Otherwise with the matching LED used, the shape really does spread the illumination evenly over the rectangle of 4x2 of pole heights, include these from such source (what would be their lifetime is a question though - shape distortion and browning over the time of heat and light exposure). The shape is not just that simple as on your picture (you have there just two basic lenses looking like these are supposed to form tow "as narrow beams as possible"), but the shape is really curved along the complete surface so it yields the desired illumination, so none of the surface section is exactly matching regular lens found in cameras or so.
I don't think the correct lens shape would be feasible to calculate analytically, it is a result of numerically adding the curve so it forms the desired beam pattern. With the computers (so converting the task from one complex equation for the whole shape into millions of simple equations of finite elements or so) it is quite easy to come to the final accurate shape and then again by using CNC's, make that shape into the mold. With traditional manual machining methods this won't be feasible, because it can not be approximated by any simpler shapes (ball, paraboloid...), for which exist the tricks to machine them.

The "Y direction" problem, i seen installers attempting to solve it, sometimes with fair result, by installing the lantern some 10 deg angled up

The "X direction" appears noticable to me even with lanterns that i am fairly sure are "new design" and dont use repurposed parts. And the road really is lit in a "square wave" pattern of bright and dim areas

But i have a question :

Assume the lantern WOULD have proper glare control, so light thrown up to 90 deg is not blinding anyone. And light distribution with an over illumination band closer to the lantern (the X direction "problem")

Why cant we then make the lanterns light up even more than 70 deg unevenly. only trying to throw the light as far as practical but not make any effort into make it even (so keep the over ilumination, just "stretch" it longer). Then the lit areas will overlap, creatng more or less evenly lit stretch ?

Indeed, there is always such overlap, it is there as a consequence of nonzero size of the light source behind the lens, but the optics is already supposed to count on the overlap so a row of lamps properly spaced forms an even illuminated stripe.

In any case, with LED's there is no reason, why there should be any overlit stripe. It's presence means mismatch between the LED's position and the lens or generally improper optical design (could be the result when one assemble the lantern by taking two panels and angling them off horizontal or so - the ready made panels are usually designed with an optic already giving complete beam pattern shape).

Then why we consider the uneven lighting from 1 lantern as bad when in a row of lanterns it does not matter ?

Then why we consider the uneven lighting from 1 lantern as bad when in a row of lanterns it does not matter ?

What matters is how a row of lights performs as a whole installation.
But given the fact farther the distance, more difficult is to control the illumination, it practically means even a single lamp (assume ideal ones) needs to provide rather uniform lighting, except the ends of the stripe, where the illumination transitions from one lamp to another. If you create an overillumination in some spot, that light can not be used to contribute to the required illumination level. The same with light spill outside of the stripe. Plus the lanterns prefferably have to cut out the light as far as possible below horizontal to prevent glare. If you take all that together, you will get a pattern, where one lamp illuminate just it's section and nothing else (glare control enforces a cut out just where it reaches the distance of 2x pole height for the standardized spacing of 4x pole height).
Therefore I have used just a rectangle being illuminated by a single lamp, such rectangles then touching each other as the poles go along the road. A bit idealized scenario...


For real life it is more practical to have the transition longer, so it becomes a bit more tolerant towards differences in the pole spacing (standard says 4x the pole height, but very often it uses to be longer and mainly it varies, according to the local restrictions like crossings or garage gates or so).

The "ends" can be as much as all of the stripe

Consider a light distribution from 1 lantern of the shape cos^2(x), appropriately stretched to match the pole distance. It is not even anywhere, but will sum up to perfectly even lighting along the entire road



Here we see 2 ways to the glare control :

1. HID way : Make the source appear large from any angle from which significant light is seen

2. LED way : Limit the glare strictly to below 70 deg

The 70 deg is still well in the visible field. It makes no sense to limit the glare to 70 deg and call that glare control : You drive along the road and every time you enter a bit into the 2L distance from a pole, a laser switches on into your eye

In the non uniform lighting way, the area near the 2L spot is lit by 2 sources, each with 1/2 the brightness of the uniform rectangle way. So the glare is allready down in half from the distribution alone, even without a change in the type/area of emitter....

Well, there is no "either LED or HID", regardless of the technology the optics has to do both at the same time, focusing the beam to cut out everything above 70deg (by the way that already means a lot of overlap for the standardized 4:1 spacing/height ratio, the 70degree cut out angle would correspond to about 5:1 ratio), as well as keep the apparent radiating surface as large as possible...

The thing is, if you want to use larger angles, the beam has to become relatively stronger compare to what illumination it gives, mainly because the road is angled towards the incident light.
If you are further away, larger area gets served by smaller beam cross-section, so if that has to carry same light flux, it means higher intensity of the beam.
To make the ratio the best (least intensity, while providing maximum illumination), you have to direct the light so, it always reaches the ground as close to vertical as possible, so each point of the road get served by the closest lantern. When you follow this really strictly, you get again the "just touching" rectangles, one from each lantern...


Of course, when you desgn the lighting with too low poles and/or too far spacing, you end up with the need of higher angle and stronger light shining there, so both results into worse glare. Or you end up with underilluminated stretches of road between the lanterns (quite frequent with HID's), but then things like a potholes or so tend to hide exactly in these spots (per Murphy's laws)...

Besides, do we really need the same Lux level in the area between the poles ? Assume it is indeed darker than under the light, but not dark to the point that you cant see anything

I think we are well capable to see in varying light level conditions, unless we are blinded by glare..

If the only criteria would be "just to see", it won't be necessary that much. Indeed, the visibility is about the ratios between the illumination level and potential glare sources.
But the light level should not change that much either, as each change needs some eye adoption and that reduces the visibility (you may see that when you leave the city to rely just on your headlight - initially you "see nothing", but after a while you get adopted)
And this ratio is the worst just in the middle between the lamps: The illumination is the lowest, the intensity for that the highest and the off vertical angle the highest.

So There the only cure is to avoid the low angles, that means not using excessive pole spacing for their height.
So when we have the optimum illumination attainable in the middle between the two lanterns, there is then no need to spend extra power for higher illumination levels just underneath the poles. So when you want to reach the visibility using the minimum light flux possible (so minimum power of the installation), you end up with even light level as well.