logoalt Hacker News

alastairr • today at 6:03 PM • 11 replies • view on HN

I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.


Replies

pkthunder • today at 6:07 PM

We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.

➕ show 8 replies
seizethecheese • today at 6:07 PM

Yeah, in real SWE tasks, 6.1 is bad.

We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)

(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)

OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.

➕ show 3 replies
hugmynutus • today at 7:34 PM

Sol-6.1 made me cancel my OpenAI subscription. Sol-5.X was bad, then 6 was barely an upgrade. Then 6.1 came out and it didn't move the bar. Went back to Anthropic.

General, SWE, python development, rust development, tool use. It is 'perfectly functional' and if you don't care about the model following your instructions or code quality it is 'fine'.

➕ show 1 reply
tezza • today at 7:43 PM

I review LLMS, and 6.1 sol is fantastic.

What kind of stuff were you throwing at it and where was it letting you down?

So what have __I__ been doing with it, then?

I’ve got 6.1-sol to take an Amiga driving game binary and add networked car driving into a web enabled Amiga emulator… that’s interpreting machine code, Amiga architecture and then making meaningful changes.

So yeah, 6.1-sol is good.

➕ show 1 reply
lnrd • today at 7:42 PM

Been using only Sol 6.1 on high/extra daily at my SWE job since it came out (with its official harness). It can basically do everything I need it to do. I still review its code and generally I have to leave around 10/20 comments in a ~700 line PR, flagging things it missed or weird decision. One or two iterations are enough to fix everything. Its still way faster than doing it myself, unfortunately.

Corence • today at 6:23 PM

Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.

https://maxxbench.unlap.ai/research/why-maxxbench/

skerit • today at 6:09 PM

I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases.

hrncn • today at 7:52 PM

I am not even certain 6.1 sol is better than 5.6 sol after using both for some time now, so not expecting much here.. but then again, 6.0 sol was completely useless, so maybe the .1 bump means something here.

ijidak • today at 9:20 PM

Dieselgate --> Tokengate (But run by the AIs)

suddenlybananas • today at 6:06 PM

You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.

aleqs • today at 7:14 PM

We know anthropic nerfs models all the time, openai probably does the same. There is no reason to believe the benchmarks/numbers they release are even measured with the same effective model that customers would get.