logoalt Hacker News

clueless • today at 5:06 PM • 4 replies • view on HN

how much benchmark-maxxing has been done here? taking bets on poly and kalshi...


Replies

xynelius • today at 5:15 PM

I think it's safe to assume every single model is benchmaxxed. I mean why wouldn't you include training data from common benchmarks? Not only is a higher score good marketing, but these benchmarks are supposed to represent common tasks so training for those should in principle improve performance across a wider range of tasks.

samseaborn • today at 5:18 PM

There's an entire section in the linked post about measuring benchmaxxing, so I guess that probably says something.

➕ show 1 reply
phoghed • today at 5:13 PM

All of them are doing all the benchmaxing they can most likely. Dumb not to when so many people care about those numbers and don’t have the opportunity to try every model out personally.

bigyabai • today at 5:13 PM

Muse Spark was generally good. I'd be surprised if this model felt benchmaxxed.