alt Hacker NewsA lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted?
What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?
Replies
IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time. https://www.intelligence.ai/ http://arena.ai
Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.
This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.
Yea and we are reaching the point where this benchmaxing is visible in the model's reported overthinking.
Have you tried it? I’d recommend doing so, it’s impressive in real use cases.