alt Hacker NewsThat suggests a straightforward mapping between context -> program. Maybe for trivial programs but I don’t see how you can argue that for sufficiently complex programs.
Reasoning models perform better than non-reasoning models because they’re able to refine their code in multiple steps. That allows any part of the program to influence any other part of the program, not just from start -> end.
Human thinking serves a similar purpose. Basically intelligence needs to be able to backtrack if you want better performance.
Right, reasoning models are almost more notable for what they don't do, which is: try to generate the first token of the response from a single forward pass through the weights.
Non-reasoning models were still surprisingly good at generating working programs. I think it makes sense when you consider that it's still iterative; it doesn't have to generate the entire import block in a single pass, for example, more like decide at the end of each line whether there are more libraries it should import or whether to move on to the body of the program. It's less surprising that a single forward pass is able to make that type of decision IMO.
Taking Python as an example, you see LLMs use a lot of scoped imports and I think this is partly because it makes the output more resilient against failure to think forward at the start. Even a reasoning model will write out (the first iteration of) an entire file in one tool call, so once it starts generating it still has to keep going until it finishes, no pausing for breath.
Also it's interesting that non-reasoning models can still do quite well in a harness that lets them test and fix up their code. Like you said, it gives them the ability to iterate.