Building an Advanced Agentic Harness

And the associated GitHub repo: https://github.com/DataForScience/LLMs

A harness [1] was developed by Terrence Tao and some collaborators to prove mathematical results. It has since then been used by others with positive effect. Can someone critique the structure of this harness? I don't know anything about this stuff.

[1] https://github.com/1stproof/batch-2/tree/main/batch-2-submis...

Any benchmarks showing if this actually improves problem solving? Or reduces errors?

The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.

So I'm a bit skeptical!

> The plan is a graph

I much prefer giving the LLM a REPL loop, and injecting all the tools as functions inside the REPL loop.

That means that the LLM isn't constrained to writing a DAG, it can write code that loops, exits early, etc.

Anyone else have related reading that touches on this? I'm building my own custom harness and want to start implementing loop support, etc. But I also want to build some sort of framework so that it's dynamic (e.g. this needs to run x number of iterations, while planning needs to run y number of iterations).

Good stuff, keeping it

why do i hate skillks, harnesses , memory systems whatever. such ideas that everyone thinks they've discovered but are totally useless in practice.

[flagged]