alt Hacker NewsSo if I try to do it with Opus three or four times, I'll get it done? And probably in about 10 minutes? Awesome
Replies
throwup238 • yesterday at 4:29 PM
That’s only if the failures are truly random and aren’t correlated
So if I try to do it with Opus three or four times, I'll get it done? And probably in about 10 minutes? Awesome
That’s only if the failures are truly random and aren’t correlated
Nope, these are no random dice rolls. Some times are solved each run, a few - occasionally (so here would be meaningful to try a few times - and metrics of pass@1 and pass@3 would be different), but most are never solved.
See e.g.: https://quesma.com/benchmarks/otel/models/claude-opus-4.5/