logoalt Hacker News

bisonbear • today at 4:44 PM • 1 reply • view on HN

These are cool results and I'm glad that tokens are being invested to eval harness capabilities.

However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.

I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)

https://www.stet.sh/blog/cost-per-successful-task


Replies

Edward40 • today at 5:01 PM

"Cost per pass" is distinct from "cost per successful task."

Cost per pass is simply the average cost per task divided by the pass rate.

If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.

➕ show 1 reply