logoalt Hacker News

Edward40 • today at 5:01 PM • 1 reply • view on HN

"Cost per pass" is distinct from "cost per successful task."

Cost per pass is simply the average cost per task divided by the pass rate.

If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.


Replies

bisonbear • today at 5:13 PM

In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail.

To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode

➕ show 1 reply