When it oversaturates with frequencies below 8 khz it just do normal audio clipping, but with sound over 8 khz and low saturation it does those weird things, like if the sampling frequency is matching random spots of the high frequency sound, like you see in the fractal looking graphic in the original post. Maybe that effect is in my case.
Well, the "fractal looking thing" is exactly, what pointed me onto the oscillation of the SD modulator. The saturation I mean is not directly visible at the decimated output, but it uses to make the output value jumping across few codes (the typical response of the digital decimation filter).
The modulator can saturate not only on absolute amplitude (input voltage too high), but as well on high dynamic change (the max voltage gets reduced at higher frequencies; exact f-V li it curve depends on the exact topology and modulator transfer function, but generally the max voltage goes down as the f increases). Usually the designs are even so the low frequency signals can not saturate the converter itself (from the analog part the upstream preamps saturate first), but usually the filters are designed so the signal gets clamped in the digital on the output. The reason is, the SD increases noise as you are approaching its ends of range, so it uses to be designed so it never operates there (a 16 bit ADC does the internal processing at least in 17 bits from -65536..65535, the output is then clamped to the -32768..32767 range of the 16 bit output).
So what may look like saturated converter means the converter itself is not saturated yet, it is just its output.
But if you put there strong HF, it may saturate the core at lower input voltages. But this saturation should be detected and the reset triggered, so it should result only into output clamping. But if this resetting does not work, i may trigger these wild oscillations (the resetting is in fact stopping it).
So I still think the ADC chip is deffective.
The chip area of this detection/reset is so tiny (we are talking about 100 gates size in the digital part and about 20 small transistor switches in the analog; compare to 100+kgates and 200 bigger transistors and 20 big capacitors in the analog for the ADC part alone; "big" and "small" is relative to the used technology, so "small" is about 0.5um^2, "big" about 100um^2), many chip makers really just play the odds of the probability of a defect being just there is very low so test only the main processing part (the parts with the "big" components) and do not test that small saturation recovery part (mainly the 20 analog transistors, as the digital uses to be covered regardless of its function just because the way how the way the test of the digital is prepared).