alt Hacker NewsOr, 2b: the nerf is real, but benchmarks are gamed and models are trained to excel at them, yet fall flat in real world situations.
Or, 2b: the nerf is real, but benchmarks are gamed and models are trained to excel at them, yet fall flat in real world situations.
I mostly stay out of the LLM space but I thought it was an open secret already that the benchmarks are absolutely gamed.