alt Hacker NewsThese general intelligence/capability benchmarks don't apply to your specific task. If you are coding, you want just benchmarks for coding. I don't see performance+price at this benchmark for coding. When you look at just coding, DeepSeek is Pareto optimal for a good portion of coding benchmarks. For example here, although even this is very flawed because it only benchmarks max/xhigh effort: https://artificialanalysis.ai/agents/coding-agents
Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.
Replies
Yes, 100% agree. Without more details on how they got their data, it's very hard to understand why they don't value DS Flash 4.1 as much as pretty much every other analysis I've seen (and my own experience).
I see this take frequently; that coding does not require general intelligence. Perhaps I am in the minority, but rarely do I find myself in coding situations that are completely divorced from other subject areas. The code is modeling or interacting with the world, either through business processes, or (in my case) simulating the physical world, and I need the agent to have some working knowledge of not just the code, but also physics, materials, or social conventions. Are you all really able to just "write code" without dealing with all the externalities?