> We study a common intermediate setting: a developer pastes one complete file into a chat assistant [5], which sees the file’s internal structure but not its callers or dependencies. We evaluate the first answer, not an agent’s result after iterating on errors.

Very unsurprising results given this methodology. The agentic approach where models are allowed to iterate on changes is a very big part of what has made LLM coding tools actually useful.