Nor is that inequality an oddity at all. If you were to think NaN should equal NaN, that thought would probably stem from the belief that NaN is a singular entity which is a misunderstanding of its purpose. NaN rather signifies a specific number that is not representable as a floating point. Two specific numbers that cannot be represented are not necessarily equal because they may have resulted from different calculations!
I'll add that, if I recall correctly, in R, the statement NaN == NaN evaluates to NA which basicall means "it is not known whether these numbers equal each other" which is a more reasonable result than False.
> "it is not known whether these numbers equal each other"
Equality, among other operations, are not defined for these inputs. NaN's really are a separate type of object embedded inside another objects value space. So you get the rare programmers gift of being able to construct a statement that is not always realizable based solely on the values of your inputs.
It's the only "primitive type" that does that. If I deserialize data from wire, I'll be very surprised when the same bits deserialize as unequal variables. If it cannot be represented, then throwing makes more sense than trying to represent it.