alt Hacker News> more like incremental belief building than "thinking ahead of time"
concretely, what's the difference here?
(I suppose you could define "thinking ahead of time" as explicitly using something like "thinking tokens" which might be roughly analogous to system1/system2 thinking, but note that we still call system 1 thinking "thinking")
Well to be fair to the author "thinking ahead of time" wasn't how they described it - that is just the HN title. "Thinking ahead" sounds like the model reasoning about the future state of the program, which is fundamentally unknown. The most the model could have learnt during RL-training, if trained in a harness, is what kind of outcomes to generally expect such as agentic progamming generally being successful (trend towards well-formed and passes tests), except perhaps for certain context smells where it appears to be going down the tubes!
My hunch is that the model isn't thinking/planning head in order to achieve these probe results, but rather that they just reflect the models current take/beliefs on "how it's going" right now, and any correlation with "how it's going" in 25 steps time is just due to the inherent biases in the setup - well-formed programs tend to stay well-formed, etc.