I keep being confused about how people's understanding of the models get stuck at next token prediction.

Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"

The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!

But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.

I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"