> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.

People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.

> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.

I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.