The core reason is because the Japanese have different numbers of red/green/blue receptors in their eyes than Caucasian people, which means that they cannot see blue light so well. Everything looks more yellowish to Japanese people, hence they appreciate natural daylight and high colour temperature light sources to be able to see colours correctly, and the colour rendering properties of light sources are far more important to them.
For outdoor lighting this resulted in an early preference for high colour temperature lamps, ie mercury. Even phosphor coated mercury was not so universally welcomed, because of its lower blue component. This trend continued long after the invention of HPS and metal halide lamps. Whereas most other countries adopted those energy-efficient sources, Japan enjoyed plentiful low-cost energy from its extensive nuclear programme and simply did not need to use more efficient outdoor lamps.
When the energy crisis happened, they had little choice but to begin developing HID retrofits that would work on the massive installed base of mercury ballasts. Hence the quick adoption of HPS retrofit lamps that had already been developed in the USA, UK and Holland. But the Japanese hated the dull orange glow of HPS and quickly began work on developing higher CCT versions based on deluxe/white sodium and metal halide technologies. The people in other countries are less concerned with colour rendering when replacing mercury lamps which are already considered by us to have poor colour properties due to their blue-white light. Hence many of these retrofit lamps remain unique to Japan.
By the way, the HPS retrofit lamps in EU were not banned without sense. It is because their life and efficacy is far lower than conventional HPS lamps and this is often further decreased since they tend to be used in extremely old mercury luminaires with poor light output ratio and inefficient old ballasts. By banning them an enormous energy reduction has been achieved in Europe, due to the need to finally upgrade those ancient luminaires to more efficient technologies.
The reason why HPS lamps require different ballasts than mercury is partly due to the fact that their lamp voltage rises during life, whereas mercury remains fixed. System efficacy is at an optimum when lamp voltage is a little over half the mains voltage, determined by the shape of the reignition voltage peaks of the arc on each half-cycle. If HPS lamps are used on mercury ballasts, their life is quite short. But regular HPS lamps that start out at lower voltages will have very long lifetimes. However, connecting a lamp of lower voltage to a ballast designed for a higher voltage lamp causes ballast overheating, premature failure, and reduced electrical efficiency. Hence a range of new ballasts having different electrical ratings was introduced with HPS lamps.