logoalt Hacker News

sams99 • today at 4:47 AM • 0 replies • view on HN

… on one benchmark, that has 2 modes of running. 1 shot and 1 shot with repair. Zero tools.

Then ignore the rules of the bench (which is a pretty obsolete one anyway), give it an agentic loop of sorts via delegation and python eyes and then win.

Sorry but this is not how benchmarking works