alt Hacker NewsAn empirical study of harness design for coding agents
Comments
Something that totally confuses me is the use of the term “harness.”
Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path?
I see both meanings used. The former is still plausible, but I see many people using the second interpretation.
Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out there (it started as just 100 lines, all included), and it's used in a lot of benchmarks like DeepSWE, terminalbench, programbench (seems like it's still top of the ranking for TB3, but wasn't evaluated with the best models on TB4).
Planning and tooling are suitably designed for execution. That's why they fail to decide on the resulting coding accuracy. If the product is user-facing (i.e. all products), a blind verification harness that is specifically designed to behave like a real user should be the decider.
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
Quite inline with what I had found with my Claude code sessions over the last year. I wrote about this a few months ago.
https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/
In their case, context management pays off more the tighter your window. Their gap between managing and not managing is 35.7 points of success rate at 32k and 2.7 points at 128k. My version of that was a rule I stick to, as much as I can. I checkpoint a session at about 25-30% of the window, write the state out to a PROGRESS.md and a JSON file of the requirements, and start fresh. This restart costs me 30 seconds, since a bloated session doesn't get any cheaper the longer you stay in it.
Also worth knowing that the models are Nemotron-3 and Mistral-Medium, not the frontier models most people here are paying for.
As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not?
I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.
nice work and paper, seeing more harness benchmarks emerge and we definitely need more. I ran mouse on the Frontier Harness benchmark and scored the highest pass rate, however I'm not convinced the results there actually translate to meaning the "best" harness in practice.
Always looking for more harness evals, although I'm going broke running them across all these models and tasks.
The conclusions:
> Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.
> Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types.
> context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets
> planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long
> structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call
Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though:
> Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default.
I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.
definitely sending this to claude to take lessons from it and audit my harness, thanks for sharing
Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back"
Anthropic appears to agree frontier models don't need in-session planning tools.
Awesome work!
I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work.
The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.)
Cool to see 1/2 of the problems keeping her on Windows solved.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
Haven't gone through full PDF as its very detailed, few things have resonated with me so far.
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.