Harness Engineering for Self-Improvement

Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.

The quest for Torment Nexus continues

Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.

One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.

After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.

I've been working with self improvement harness a little bit and one thing i've come to conclusion is harness task fit. The learning can be significantly improved if we understand the behaviour of task and how it should be learned. I'm pretty sure a general solution will definitely exist which will do fine, but we are yet to see one.

They say engineering but it's more a soft science.

The simplicity is the point

one form of very effective self-improvement that coding agents do all the time:

install or build stuff that they can then use

it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?

animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'

[dead]

[flagged]

[flagged]

[dead]