logoalt Hacker News

embedding-shape • today at 6:45 PM • 0 replies • view on HN

> There is always a huge discrepancy between benchmarks and real world capabilities.

I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.