alt Hacker NewsThe meat of the report for SWEs:
SWE-Bench Pro Sol: 64.6% Fable: 80% Opus: 69.2% (!!!!)
So, it still trails Opus, significantly, and is not a next-gen coding model like Mythos/Fable 5.
Disappointing to say the least, but somewhat expected.
Replies
sk4rekr0w • yesterday at 5:21 PM
You've overstated the conclusion. The SWE-bench series has had issues since its inception.
OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...
➕ show 1 reply
paxys • yesterday at 5:19 PM
Makes sense why they released an entire study yesterday discrediting SWE-bench Pro.
➕ show 1 reply
CuriouslyC • yesterday at 5:21 PM
[dead]
SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.