logoalt Hacker News

billyloyesterday at 3:02 PM2 repliesview on HN

That's why benchmarks are useful. We all suffer from the shortcomings of human perception.


Replies

gpmyesterday at 3:09 PM

Benchmarks shortcomings are no worse... they inevitably measure something that is only close to the thing you actually care about, not the thing you actually care about. It's entirely plausible that this decreased benchmark score is because Anthropic's initial prompting of the model was overtuned to the benchmark and as they're gaining more experience with real world use they are changing the prompt to do better at that and consequentially worse at the benchmark.

show 1 reply
turnsoutyesterday at 3:37 PM

Benchmarks measure what they measure. But your subjective experience also matters.