logoalt Hacker News

onlyrealcuzzo • today at 8:08 PM • 1 reply • view on HN

There's thousands of benchmarks, and nobody can agree on what the best models are if you run all of the benchmarks which costs tens of thousands of dollars for frontier models.

Then, somebody runs a small subset of benchmarks - with not enough data to have any statistical significance - and makes further claims.

How are we supposed to derive value from this?


Replies

Edward40 • today at 9:15 PM

Although people disagree with the best models, the worst model is obvious.