logoalt Hacker News

seizethecheese • today at 6:07 PM • 3 replies • view on HN

Yeah, in real SWE tasks, 6.1 is bad.

We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)

(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)

OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.


Replies

locusofself • today at 6:27 PM

I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me

➕ show 2 replies
throwitaway222 • today at 6:50 PM

I'm guessing you don't use codex though- the harness + model combo makes a difference.

➕ show 1 reply
slekker • today at 6:24 PM

Is this an ad?

➕ show 2 replies