logoalt Hacker News

What is the best harness for each model?

296 points • by Edward40 • today at 2:53 PM • 105 comments • view on HN

Comments

seizethecheese • today at 4:36 PM

The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers.

It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.

Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".

Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.

➕ show 7 replies
reversengineer • today at 9:51 PM

Is it me, or do the results seem to show a lack of significant difference between both models? And, because of the potential non-determinism in Generative AI to create a huge degree of variance in outcomes, I am dubious about the repeatability of these results.

Zigurd • today at 4:12 PM

I find myself pulled in two directions by the performance of coding agents: First, keeping them on a short leash seems sensible. I don't get failed results very often. When I first started using coding agents, this seemed not just sensible but required in order to not deal with a lot of hallucinated weirdness. Secondly though, it's pretty cool to vaguely wave your hands in a certain direction and get both user interface and business logic that's objectively nicer than I would've gotten from the effort I would've put into manually coding. Whenever I cautiously use a coding agent, I wonder what I am missing by not letting it cook.

➕ show 6 replies
xyzzy_plugh • today at 5:52 PM

I don't like this framing.

For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.

Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.

This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.

But I can't build a mobile app with this harness, or make a movie.

These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.

Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.

Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.

➕ show 2 replies
cpeth • today at 8:36 PM

Harnesses are all fairly simple loops from the ones I have looked at; there is not much to them.

As someone who regularly uses Claude Opus 5.5 from Claude Code and from Copilot, I would say harness barely matters in terms overall performance and behavior. Opus is nearly identical in both harness. The opposite is also true, if I switch Copilot to something like Gemini 4 Argon it's completely different even though I'm in the same harness.

Just some andecdata.

➕ show 1 reply
bisonbear • today at 4:44 PM

These are cool results and I'm glad that tokens are being invested to eval harness capabilities.

However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.

I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)

https://www.stet.sh/blog/cost-per-successful-task

➕ show 1 reply
ford • today at 4:29 PM

To be clear - "Failed" here means "Did not meet all pass criteria on a given benchmark task", not "Experienced a runtime failure"

zodwick • today at 8:42 PM

I wonder if there is corelative improvement between the same lab's own harness and model.

For example I remember Codex having a different line number format than CC and every other harness had to have a two format tool to maximise performance.

I wonder if sucha thing exists still or do they RL on all possible combinations now

alok-g • today at 8:50 PM

Genuinely asking: Why is Kilo code (https://kilo.ai) not present in this and most such analysis? It does not also get talked about much on HN in general.

scosman • today at 7:41 PM

We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.

I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.

scubazealous • today at 4:32 PM

While this could be better, I wonder how much time the average developer spends on runs that fail? How much time do students spend on runs that fail?

fennecfoxy • today at 4:51 PM

Yeah, developers sometimes go down the wrong path as well. It's a non deterministic system exploring a huge problem space...

➕ show 1 reply
franzb • today at 8:45 PM

Original title: "In 27,280 coding agent runs, 64% of inference spend went to runs that failed"

elAhmo • today at 6:51 PM

So, on my daily coding tasks choice of provider, model family, model version, effort/thinking level and harness all have an effect. Some of these are multiplicative (like harness), so this is all extremely confusing for the end user about what the hell should I use do do my tasks.

evalstate • today at 5:13 PM

Just use a harness that has been properly integrated and tested with each model, with published traces and comparisons (author of) https://fast-agent.ai/benchmarks/

GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.

d-sanoj • today at 8:05 PM

I think the harness matters less when we are measuring the work done. There are all kinds of benchmarks in differ cases showing one model is better than the other. The existing ones like Claude Code or codex work well with their models.

cronin101 • today at 4:22 PM

I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)?

Can anyone confirm/deny this is the case?

If so, it seems a shame not to include this practice into the reporting methodology for full transparency.

➕ show 4 replies
SAI_Peregrinus • today at 6:03 PM

When a task passes all tests, it's done. More tokens should be spent on failing runs, because most runs where tokens are used should fail. For the same reason that your lost keys are always in the last place you look for them.

Majromax • today at 4:05 PM

Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps.

The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.

I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.

➕ show 1 reply
samueloph • today at 5:25 PM

Am I'm missing something or at this point the distinction between "harness" and "agent" doesn't exist anymore because too many people use them interchangeably?

Is this post mixing them up or is my understanding of these concepts off?

➕ show 2 replies
augment_me • today at 4:55 PM

This is completely expected. Agents are pseudo-greedy search functions that brute-force solutions using their context.

If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.

➕ show 1 reply
bensyverson • today at 6:58 PM

This is interesting and valuable, but I'd love to see a follow-up that tests a more parallel or subagent-heavy workflow.

jrmg • today at 4:25 PM

"In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title.

The page's title is: "Introducing FrontierHarness Eval v1.1"

➕ show 1 reply
andai • today at 4:39 PM

>Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.

I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.

And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)

bob1029 • today at 5:05 PM

The entire point of this AI thing is that the cost of failure is now ~zero. You can try again in a few minutes / hours. A traditional software business might only complete one iteration before time runs out.

➕ show 1 reply
colinmarc • today at 4:41 PM

How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.

➕ show 1 reply
WorldPeas • today at 5:14 PM

evolving harnesses are also a cool development I've seen recently though I feel they'll have diminishing returns https://www.alphaxiv.org/abs/2609.01437

justincormack • today at 5:34 PM

You do need to optimise prompts too, eg I found that openai models just gave up in Pi, but adding a line to the prompt matching Codex eg "Once implementation is authorized, the agent should persist through the bounded change, tests, review and final validation. Complexity, expected duration and context compaction alone are not reasons to stop." fixes that. Otherwise it would look at tasks and say they were too hard.

winddude • today at 6:29 PM

Reading a lot of the comments everyone is talking about state actors, etc.

Let's not forget multiple departments like the Pentagon, FAA, Homeland Security etc, have been raising alarm that things like this were becoming a heightened risk since Trump and Musk cut ~300k jobs with DOGE, and slashed civilian contractors across nearly every department.

BugsJustFindMe • today at 5:34 PM

This is nice and all, but I really want to see the best harness for the actual frontier models ChatGPT and Claude, not just open weight models.

worik • today at 6:33 PM

Given the variation between results for different, but similar capability models it is hard to take this seriously

maerF0x0 • today at 5:00 PM

So? I'm sure exceedingly high amount of EC2 cpus are idling, or lost to discarded branch predictions etc.

We waste whatever is cheap to gain what is dear.

(Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)

0xbadcafebee • today at 5:32 PM

The choice of agent/harness is, as people frequently point out, just one of many variables that changes agent performance. Skills, MCPs, Extensions, system prompt additions, often have drastic effects, making things better and/or worse. It's interesting to compare them side by side, but probably won't reflect your own unique experience.

Second thing to consider: ZCode was sending everyone's code to Alibaba... not sure I would use their harness just to save a few cents on coding.

jedisct1 • today at 5:18 PM

It would have been nice to see swival.dev in that list.

But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.

brookritz • today at 6:24 PM

The TLDR saya: Basesten, Harneas and Leaderboard and still have no clue.

shevy-java • today at 4:39 PM

Well, it is a milking system. You have to pay for those tokens anyway.

➕ show 1 reply
underdeserver • today at 5:00 PM

Wait till you hear what the human success rate used to be.

aleqs • today at 7:04 PM

Another bullshit result obviously meant to promote a specific company/product. Codex is garbage btw.

guanlan • today at 2:57 PM

Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.

The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x

What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.

What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.

➕ show 2 replies
johnrc • today at 6:44 PM

[flagged]

amarinhodev • today at 8:48 PM

[dead]

shiqimei • today at 2:56 PM

[dead]

Olscore • today at 6:49 PM

The harness doesn't matter. You need a leash.

Once you have sufficient organization around the agent/harness you start to see how a harness should be designed (for the leash.) Everybody seems intent on building a better PR-loop, or a better way to drive an IDE. To me this is backwards; I'm trying to change it. No IDE. No PR-loop. Outcome only.

https://github.com/pullboard-dev/pullboard

➕ show 1 reply