Artificial Analysis v5 and the benchmarker's dilemma

I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.

I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.

We just need transparency. It's okay if you change criteria, but pre-register the trigger for when you're going to change them.

Example could be, we'll update X criteria when top-N models are within 2 points of each other.

"The final point I need to make, which does flatter the closed labs, is that the likes of Anthropic and OpenAI moved on from caring about these indices months, or even years, ago. Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars."

Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.

Mentioned in the article: https://www.intelligence.ai

Not sure if the scores are better, but the website is nicer to use.

AA has such terrible UX, especially on mobile, trying to select a subset of models you care about is an exercise in frustration.