I put up my SOX 18w wallpack in my tiny backyard.

Turns out those things actually are pretty damn bright. Definitely gonna attract attention. I've had it lit indoors on occasion but somehow to the eye it didn't look particularly powerful, but outside in the dark it's just a sea of yellow, brighter than the cone top public street lights which are pretty dim here because you don't want those to shine into your bedroom window either.

I'm gonna make a little shroud for it so it doesn't shine into the houses opposite mine.

The principal solution is to not "optimize" the lamps by reducing the heatsink size and number of chips in a lamp of the same power rating

Max power depends on initial conditions and room size, while heat loss is insignificant at that stage. Steady state depends only on the heat loss

As unit size is chosen mostly based on its max power (to get a room from initial conditions to desired temp in wanted time), this means that the same unit may be installed in rooms with same initial conditions but very different loss

Still interesting what the real reason is! I believe that CO2 is quite reactive inside a lamp. At least, even in deliberately badly made cold cathode tubes it is consumed by electrodes in some hours of burn-in. Not 100% sure for hot cathode, but can expect CO2 will dissociate in the discharge and react with mercury vapor, precipitating elementary carbon and mercury oxide rather quickly.

Currently, I have a 400W PSMH Westinghouse lamp made in China, and I also have a GE M400 in 400W HPS, which are obviously two different lamp technologies. Recently, I was wondering what would happen if I ran the 400W PSMH lamp on the 400W HPS ballast. The arc voltage drop is higher than that of a 400W HPS lamp, so how would this affect the ballast? Would it overload the ballast?

Is your 400W ballast of HX or CWA topology?

The "turning purple" is just one particular failure mode on which just the particular LED design used in the streetlights was prone to. I don't think that style is used anymore, not just because of the tendency to turn purple, but also because the LEDs are extremely vulnerable to damage during assembly.
For domestic LEDs a broken bonwire failure (LED chaotically flickering or cycling, then dying, often with a noticeable internal burn mark) is the most more common failure mode these days.
It seems most problems become solved, except the bonwires. It seems there is no principal solution except some minor optimizations here and there.

It is matter of selecting particular unit power size for the given task. Along with the maximum power, to ensure it would be able to keep up with the mose demanding condition.
Obviously you won't fit exactly, but you should aim close, selecting a model/rating variant that is the closest.with its performance.
Practically it means how the manufacturer matches the exact performance with the normally used power rating category and the climate of the target market area (each exact AC designs are tailored for the given territory separately, combination of the temperatures, winds, humidity and so on, so a unit sold in one place may perform very bad somewhere else just because it is set for different climate).

I have seen the opposite thing happen once

The diffuser material on the LEDs ("1W"/"3W" style gull wing LEDs) degraded from the intense blue content in the light, discoloring to a burnt brown color and making the light color shift to some orange-ish color as well as drop in brightness

This happened in lamps that been some 12+ years in service though, most lamps just dont last anywhere this long

The level at which the unit must run (including above minimum) would vary with each individual installation. It is not something the unit designer can predict with high precision

I second Multisubject's answer

There are PSMH lamps for the European market rated for this exact use (400W HPS ballast), but there the ballast is just a choke so there is no primary winding to overload

500W is a significant overload, so unless the ballast is running in a significantly reduced ambient temperature (able to compensate for ~20W ish additional loss), this will probably lead to overheating of the ballast

A ballast of this size has high thermal capacity, and may take well over an hour to stabilize at steady state temperature for measurement