The LLM doesn't produce erroneous output, its generated tokens fall within the statistical limits of the preset configuration, unless some kind of bitflip messed up the model in memory somehow. An LLM doesn't tell the truth or answer a question, it just spits out tokens. Its training doesn't involve validating whether or not the output forms a true fact or statement, but rather if the output looks like one. For the same reason, an LLM cannot lie, because an LLM doesn't have any intention, nor can it tell the truth. That level of thinking is beyond the capability of an LLM.

The term "hallucinations" are an anthropomorphised interpretation of valid output that's factually incorrect. It happens to people all the time (the human brain will make up any missing memories and subconsciously explain away inconsistencies, which only becomes obvious once you're dealing with someone with memory problems), so it feels like a decent term to use for garbage information produced without any ill intent.

The problem lies with the AI companies convincing their customers that the output generated by their tools is probable to mean anything. Probability engines are sold as some kind of chat program or even as some kind of autonomous agent because the output comes close enough to pass the Turing test to most people. LLMs can only mimic intelligence, interactivity, or any other kind of behavior, they cannot actually think or reason.

If people knew what they were operating, the "hallucinations" wouldn't be a problem. Unfortunately, that would take out most of the confidence people have in these tools, so you won't see the AI salesmen provide their customers with reasonable expectations.