logoalt Hacker News

seizethecheese • today at 4:36 PM • 7 replies • view on HN

The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers.

It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.

Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".

Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.


Replies

seizethecheese • today at 4:45 PM

Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere!

A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.

➕ show 4 replies
onlyrealcuzzo • today at 8:08 PM

There's thousands of benchmarks, and nobody can agree on what the best models are if you run all of the benchmarks which costs tens of thousands of dollars for frontier models.

Then, somebody runs a small subset of benchmarks - with not enough data to have any statistical significance - and makes further claims.

How are we supposed to derive value from this?

➕ show 1 reply
Edward40 • today at 6:08 PM

Good finding! Model preference is interesting.

Edward40 • today at 4:49 PM

Because these tasks are genuinely hard and require long-horizon reasoning, running full evaluation experiments gets expensive fast.

For example, on bun-sourcemap-leak (https://hub.harborframework.com/tasks/terminal-bench/bun-sou...), all 440 trials finished cleanly with zero infra errors or timeouts, yet 0 out of 440 passed all hidden-variant tests (the best was 35/36 across three DeepSeek V4.1 Flash runs).

Some of these tasks take up to 8 hours per run, so if you are not careful, you end up burning massive amounts of compute and API spend just to get an accurate evaluation signal.

➕ show 1 reply
plufz • today at 4:46 PM

Yeah exactly, compared to what baseline? People here will probably be stunned when I say this, but my pre-agent code was usually not bug free on my first attempts either.

themgt • today at 4:59 PM

Over the 22 harnesses, the rank correlation of the steps per pass and the cost per pass was +0.51 (p = 0.015). For the other three factors, it was −0.03 to +0.02 (p ≥ 0.88). Within each model, tool calls tracked cost as well as steps. Table 4 below and Table 5 in the appendix give the values.

For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.

Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!

➕ show 1 reply
adrithmetiqa • today at 5:46 PM

Agreed. Does this help anyone decide what model or harness to use. Nope.

➕ show 1 reply