Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.

[0] https://deepswe.datacurve.ai/

[1] https://cognition.com/blog/frontier-code-1.1

I've generally found DeepSWE[0] to be pretty true to reality.

[0]: https://deepswe.datacurve.ai/

https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)

strawberry

FrontierBench