alt Hacker NewsLLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language, as opposed to a property of "mind" (either biologically or artificially instantiated.)
LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.
> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.