Towards a Harness That Can Do Anything

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.

Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.

I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.

Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.

How much do the labs post-train on the harness inputs & outputs? That's a critical piece to understand if a "generic" harness is at all possible

Awesome work! This is really impressive. I gave a GitHub star.

I build precision-editing tools for AI coding agents (hic-ai.com) and worked out thousands of JSON-wrangling and regex issues, so I can verify they are indeed a bit of a pain, across all possible failure modes that AI coding agents and models and harnesses can produce. Anyway, I completely agreed with everything in your article, though I would suggest however that agents need *three* things at runtime to fix a defect: great logging and a clear error response (just like you have it), but also, precision-editing tools that enable agents to make the minimal, surgical change without touching or copying any other portion of the file. These actually change not just the feedback but also the options available to the agent and capabilities in the midst of the workflow to self-heal. If Ambiance adds a kernel to buffer the LLM from the outside world, HIC Mouse adds a "kernel" or buffer between the LLM and its own environment and file system. Anyway, this is such a cool project. Please reach out if you ever add MCP support for Ambiance -- I'm happy to release a new version of Mouse that supports it. Again, great work.

What has been the most helpful when developing harnesses:

> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.

I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.

What's with this "harness" word people have been trying to adopt lately? Are we all going rock climbing?

Why are we not just using Claude Code or Codex on our machine and using this thing? Real question...