logoalt Hacker News

megavontoday at 5:16 PM1 replyview on HN

Need to look at SWEBench-Pro, it's super competitive. Suspect they'll catch up given the longer-tail on TB scores.


Replies

jaentoday at 5:33 PM

Just by the (lack of) inter-model variance, I don't think SWEBench-Pro does a very good job of representing model capability. Terminal-Bench seems more challenging and separates the wheat from the chaff.

Also, *ops work, which in my experience can actually be more complicated than SWE is underrepresented there obviously.