They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case.

I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.

Sisyphus doesn't care about energy per meter to move the rock.