logoalt Hacker News

meatmanektoday at 5:53 PM1 replyview on HN

I just want to rant about these Artificial Analysis charts that you see everywhere:

The "most attractive quadrant" is completely meaningless. The whole point of a Pareto curve is that each point on the curve is better than everything else on at least one dimension, and that you can make these comparisons without placing a value judgement on the relative importance of the different metrics. If you make a composite score of the two metrics (any monotonically non-decreasing function, e.g. a weighted sum with non-negative weights), that score will always be maximized by one of the points on the Pareto frontier.

So going by the numbers in the 2nd chart (1st AA chart) from TFA alone:

   - there's no reason one would choose Deepseek V4 Pro 0813 (max) even though it's in the "most attractive quadrant", because GLM-5.3-Flash is both cheaper and scores better.
   - Claude Fable 5.1 (max with fallback) on the top right* could be your most attractive option if you need the best scoring model and don't care about cost, even though it isn't in the "most attractive quadrant"
   - The un-shown model off the left side of the chart could be your most attractive option if you just need lots of cheap tokens and don't care about quality.
(Obviously if you start including other factors in your score that aren't represented on the chart, then you might choose differently.)

* I also dislike the way they place the labels, and that grey line connecting the label to the point is way too subtle.


Replies

Hugsuntoday at 8:13 PM

Sound critique. I'll add that the Artificial Analysis intelligence index is not considered a good metric for intelligence anymore. Most of the benchmarks that it comprises are saturated or considered low signal today.