MaxxBench – "Nice DeepSWE score. But is it true?"

Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.

The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.

Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.

All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.

The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.

Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.

(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).

If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.

edit to add: Here's a rough scatterplot to explain what I'm seeing ("(?)" means parameter counts are community estimates),

https://files.catbox.moe/gryjzm.png

One thing to consider is that even when some model has big gap between those two benchmark it doesn't mean it is much worse. It could be worse becasue he has assuptions based on training but on real scenario it wouldn't have it so it still could achive great results

This is perfect. Real programming has a ton of these instances of minor changes.

DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.

I liked the GLM models until 5.3 and this site seems to explain perhaps some of the reason why.

[flagged]

> Most models tried pleasing an imagined grader over the prompt

>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.

I ran into the same problem on my English exams.

It's a pity it doesn't show GPT 6.1 Sol and Terra or 6 Luna.

There is always a huge discrepancy between benchmarks and real world capabilities.

For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.

So basically it turns out the good models are good and the benchmaxxed models are benchmaxxed.

Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.

https://news.ycombinator.com/item?id=50406948

But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?

HN hug of death victim?

Two brothers created a benchmark, family is so proud.