> So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful.
Only if one believes the logical fallacy that the dependent steps of a process of elimination weren't useful.
Eh, in many ways the problem is a sunk cost fallacy type issue.
If someone is later in their career and looking at having to throw away all that time - time they will never recover - it takes someone really special to just do it.
And by really special I mean ‘kinda suicidal sometimes’.
Even if you believe that they are useful, you're also not going to wind up as a hero in the history books. And so people wind up acting in the same way.
Besides, the argument that all of the bad ideas contributed to discovering the right one, is as strong as the empirical argument that white chairs are evidence that all ravens are black. Logically you're right. Discovering the right idea requires disproving all of the wrong ones. Similarly "all ravens are black" is logically the same as its contrapositive, "all non-black things are not ravens". It's just that you've just decided to focus on a search space that is so much bigger, that each data point in it becomes much less important.