The fundamental reason for this is simple. Humans are prone to cognitive dissonance. Meaning, we do absurd things to avoid painful thoughts. And anything that questions our sense of identity, is a painful thought.
So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful. So we avoid thinking it, challenge people who question our past contributions, and so on.
The natural result of this cognitive dissonance is a feeling of undue certainty in our speculations. After all certainty is merely a belief that one idea is easy to believe and its opposites are hard to believe. We imagine that our certitudes are based on fact. But they more easily arise from cognitive biases.
And this is how a group of intelligent and usually rational people descend into theology whose internal contradictions can't be acknowledged.
To extend this, group dynamics can come into play too.
I once worked at a startup that developed fancy new tec. The group dynamic there was that critical thinking absolutely did not exist. The reason was probably that they accepted only people in their circle, that had the same burning positive attitude towards the idea.
This can become a self reinforcing circle, because critically thinking people will leave at some point. (Like sabine did in physics).
> So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful.
Only if one believes the logical fallacy that the dependent steps of a process of elimination weren't useful.
This is where climate skepticism comes from by the way. Even climate skeptics will acknowledge that climate scientists are well educated, they don’t deny science as a process of truth seeking, the problem lies in the incentives.
There’s a lot of prestige and grant money that comes with insisting climate change is true.
There’s a lot of political power that gets ceded to the people in charge if we “just accept that we’re in a crisis and us elite are the only ones that can stop it”.
I believe climate change is real and human caused, but many of the claims and doomsday speak feel like self interested humans following their incentives beyond the scientific truth
This is beautifully articulated.
And reinforces my general below-the-line (layperson) fear about the state of physics today (as reinforced ofc by the likes of Sabine Hossenfelder & Eric Weinstein).