logoalt Hacker News

seizethecheese • today at 5:34 PM • 0 replies • view on HN

> MAI Code 2 reaches 98% of the coding quality of GPT-6.1 Sol at about 40% of the price.

To me, this smells like the same trick I always see in claims about model routers. As in, running against a saturated benchmark then showing huge cost savings for a tiny performance drop (because almost any model will perform well).

Looking deeper, though, it looks like they are citing four benchmarks, only one of which is basically saturated (Terminal Bench 2.1).

On the other hand, on 2/4 of these benchmarks Sol 6.1 does better than Opus 5.5, which just doesn't pass the sniff test, so my conclusion is that these are cherrypicked benchmarks.