Cognition's SWE-2 achieves 92.8 on Terminal-Bench 2.1

Terminal Bench 2/2.1 appears to be almost solved, so we probably shouldn't look too hard on that? Their Terminal Bench 4 score is soso.

Still quite impressive though.

my personal experience with swe has been…suboptimal. I am not sure how much I buy these benchmarks and it has a very “just blurt it out even if it’s probably not right” style but hey it’s free.

Not really news, terminal bench 4 is the new metric. It's only a few points ahead in terminal bench 4 of some open weight models you can run on a 256GB system.

Related:

Cognition launches new SWE-2 model

https://news.ycombinator.com/item?id=49645443

These tests are pointless when often models get nerfed few days after release. Astra today is way dumber than just few days ago.