alt Hacker News> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.
I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.