alt Hacker NewsInteresting. Framing it as 1x to infinity-x matches my experience too. I've have good success with it reproducing papers with existing code, but not such much with one-shotting new code. Do you have any particular setup for this i.e. special validation prompts or multi-agent stuff or do you just ask something like reproduce paper X until it gets results Y?
I used to chat with "the paper" until both me and the AI were roughly on the same page at what's happening. It's important, as a "full read" of the "full paper" is not always used efficiently if not discussed, I've noticed. I then ask it for a detailed step by step realization (of the algorithm, or the paper/section/whatever), plus some context. Then I pass this as a prompt to a fresh new instance, and task it to implement in full based on the description (Gemini Deep Think was quite good with that).
Now I feel like 5.6 Sol Ultra is capable of doing roughly the same with its Agents, so it's getting easier in my experience. With the Codex, it can adjust or correct until the output is suitable.
I'm sure it depends on the field and the method.