If I could change one thing in computing, it'd be how SQL handles NULL. But if I got a second thing, it'd be how IEEE handles NaN. I probably wouldn't even allow NaN as a representation. If some mathematical operation results in what would be NaN, I'd rather force the programming language to throw some sort of interrupt or exception. Much like what happens when you divide an integer by 0. Heck, I'd probably even stop infinity from being represented with floats. If someone did 1/0 or 0/0, I'd interrupt rather than generating an INF or NaN.
In my experience, INF and NaN are almost always an indicator of programming error.
If someone want's to programmatically represent those concepts, they could do it on top of and to the side of the floating point specification, not inside it.
Floating-point infinity is actually really useful in Python because you can easily and efficiently compare it to Python's arbitrary-size integers.
People who criticize the IEEE standard typically have never read it.
Nobody forces you to use NaNs or to have partially-ordered floating-point numbers.
The default settings that have been recommended since the beginning by the standard were to not use NaNs and to have totally-ordered FP numbers.
For this choice, the invalid operation exception must not be masked.
However, it turns out that most programmers are lazy and do not want to bother to write an exception handler, so the standard libraries of C and of most other languages have chosen to mask by default this exception, which causes the generation of NaNs.
This bad default configuration in combination with the bad feature that most programming languages have only the 6 relational operators sufficient for a total order, instead of the 14 operators required for a partial order, makes necessary frequent tests for detecting whether a value is a NaN, to avoid surprising behavior in relational expressions.
I think that in most cases the extra checks for NaNs add up to much more work than writing at exception handler, so the better solution is to unmask the invalid operation exception, which makes the NaN problem go away.