SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.
But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.
alt Hacker News
SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.
But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.