alt Hacker NewsTo be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Replies
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
Blog articles from Anthropic and others show that this is not true.
A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.
There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.
Maybe I'm wrong - but I see LLMs are a "next-token predictor" as somewhat equivalent to brains are a "bag of molecules".
Both systems have emergent behavior that goes well beyond what would naively be expected.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".