alt Hacker NewsSWE-bench Verified was created to fix the problems of SWE-bench.
Then SWE-Bench Pro was created because SWE-bench Verified had flaws.
Now SWE-Bench Pro is shown to have flaws.
Replies
denysvitali • yesterday at 9:50 PM
Well, we now have DeepSWE
SWE-bench Verified was created to fix the problems of SWE-bench.
Then SWE-Bench Pro was created because SWE-bench Verified had flaws.
Now SWE-Bench Pro is shown to have flaws.
Well, we now have DeepSWE
Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?