logoalt Hacker News

anana_ • today at 3:03 AM • 4 replies • view on HN

Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1


Replies

rbehrends • today at 3:45 AM

While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.

[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.

➕ show 1 reply
anon373839 • today at 3:07 AM

AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.

Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.

➕ show 3 replies
selectodude • today at 3:05 AM

Anthropic finally figured out how to block them from distilling Opus.

➕ show 1 reply
AAisthetiobeofA • today at 3:26 AM

> Artificial Analysis

Ah yes, the reverse-benchmaxers of OpenAI models.

➕ show 1 reply