I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.

But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.

In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.