alt Hacker News"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
Replies
But it is a next token predictor.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
Good example.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
> used most of the time with a motive to downplay the abilities and faculties of AI models
Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
That's literally what LLMs are.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.