alt Hacker NewsAre benchmarks the right way to measure LLMs? Not because benchmarks can be gamed, but because the most useful outputs of models aren't things that can be bucketed into "right" and "wrong." Tough problem!
Replies
olliepro • yesterday at 6:40 PM
Do you have a better way to measure LLMs? Measurement implies quantitative evaluation... which is the same as benchmarks.
➕ show 1 reply
Not an expert in LLM benchmarks, but I generally I think of benchmarks as being good particularly for measuring usefulness for certain usecases. Even if measuring LLMs is not as straightforward as, say, read/write speeds when comparing different SSDs, if a certain model's responses are consistently measured as being higher quality / more useful, surely that means something, right?