I recently needed a bright light for a few hours and decided to take out my magneticly ballasted 400W MH fixture, with a custom ballast enclosure. After using it for a few hours, I measured the temp of the ballast and it was 168 degrees Fahrenheit. Is this too hot? I have recently added a fan to the enclosure but I wanted to see if that would have damaged it.

All of the ballasts I have also get quite hot, I feel like it's normal but I didn't measure any actual temperatures. I have a feeling this is partly due to using thinner copper/aluminum windings to save money but also partly due to the core layers being welded together on almost all ballasts, resulting in eddy currents that heat the core itself. It probably wouldn't have been damaged, but I would measure the wattage of the ballast to see if it is really high. I hear that significantly above 500W input would be not so great, though I could be wrong.

Is this too hot?

No. 168F equals to circa 76C which is rather benign for a magnetic ballast.

Caveats:

1) Modern ballasts usually have Tw and deltaT ratings for winding insulation thermal capability. That should give you an idea of maximum winding temperature allowed.
2) As you see, thermal ratings are given for winding insulation temperature. Internal temperature is somewhat higher than you measure on the ballast surface.

The deltaT will give you ballpark what operating temperature to expect as normal (you add it to the room temperature).
If you then measure something way different, it means the power dissipation is different than expected, so something does not run normal...