I think NaN itself is a bit of an error object, especially in how it's passed through subsequent math functions, which is a different choice than throwing up.
But besides that I think you're right, Invalid Date is pretty weird and I somehow never ran into it.
One consequence is you can still call Date methods on the invalid date object and then you get NaN from the numeric results.