logoalt Hacker News

aenis • today at 9:32 PM • 0 replies • view on HN

Big progress, for my types of workloads.

When given my benchmark suite, which involves a couple of long horizon tasks where the agent decides when to consider the task well executed, its the first OpenAI model that does not want to finish as soon as some minimal level of quality has been achieved. Like Fable, Opus and Sonnet it now runs the maximum allowed iterations and does not stop at 'satisfactory', gold plating the solution to my delight.

(I don't work like this on work projects, but for everything hobby, I am testing how much autonomy I can give to the coding agents, and thats what my benchmark suite tests; I have the basic suite which completes in 2-4hrs, and the long running suite which takes about a week or so; the other one I just triggered, but the first iterations look way better than prev gen Sol/Astra at the same effort level).