If you use -ffast-math when linking an executable but not a shared library, both gcc and clang will link in crtfastmath.o which has the bit of code to set the DAZ/FTZ flags.