alt Hacker NewsI try to make 3 claims in the post, it was a bit clumsy I'll admit that.
1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that.
2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what the ground truth next token will be in the data.
3. During post-training, in the case of RLVR, there is no ground truth next token. In pretraining, the question is "what token actually came next?". In RLVR, the question is "what sequence of actions gets a high reward?"
And the whole point is that thinking about the RLVR is important. A mental model that stops at 1 or 2 is incomplete and doesn't capture what drives LLM tokens.
Probably the easiest way to describe an LLM that it's a policy. There is a reason that word has stuck in RL.
And it's not just RLVR. RLHF has been going on for years and years. LLMs have not been "next token predictors" for probably 5-6 years.