It's language and speech patterns that seem designed to trick readers into believing that claims are correct, even when the claims aren't based on anything and are possibly wrong.

It was rewarded for this during training for some reason.

Alternative theory:

The LLMs only way to "think" about abstract concepts is through language, and this leaks into into conversation it has with humans.

But humans generally prefer to communicate on low levels of abstraction, through a back and forth, until the hard-to-express higher abstraction exists in the head of everyone involved - without ever being directly communicated. This is because we don't think using language. Language is merely a lossy translation of our thought into something expressible, happening after the fact or alongside it.

So when the LLM starts speaking to us using patterns and terms it created for itself during training to encode abstract thought in language, communicating with it becomes painful.

You can thank everyone who thumbs up this response along the training pipeline.

Modern LLMs are not too different from Reddit/Twitter in that regard, I'm sure the AI labs learned (lol) a lot from them re: how to do "engagement".

This matters. That's the spine of it.

the phrase load-bearing caveat makes me irate