Yes Hg is very toxic and can cause brain damage this was the inspiration of The Mad Hatter in Alice in Wonderland as mercury was used in hat making, many barometers (not aneroid) have mercury which is open to air so they work properly Torricelli affect so make sure there is good air flow at all times.

It is not the elemental mercury, but its compounds, what are extremely toxic. With hat making process it was in the bath the felt was treated...
Elemental mercury (which is found in lamps) toxicity is only mild, because it tends to condense and form liquid globules, which have rather limited surface to react into something nasty.
Elemental mercury is dangerous only as gas when it could be breathed in (when it is way more reactive due to its ability to mix with other reactants), but being gas needs sufficient temperature to reach high enough concentration

Plus you need sufficient dose to do some harm. A single lamp is way not enough, but e.g. material storage at manufacturing plant or an open landfil filled with 1000's of old tubes may bring way enough exposure...
But e.g. a mercury rectifier room in a tram supply station with raging fire is able to kill you after few seconds of exposure - accident that happened about 50 years ago, a fire fighter entered the burning rectifier room without thinking about the possible hazard there - and there were kg's pools of mercury from the broken rectifiers boiling, died within days...