Yesterday I started working on an agent harness that tries to address some of the issues here.

What I'm hoping to build ultimately is something that works more like a pair-programming partner than existing harnesses do. I want the user to be an engaged part of the development process all the way through, I don't want the agent disappearing to work on its own. I even want to make it possible for users to swap into the driver role and have the LLM automatically assume the role of navigator when that happens.

There's more info in the readme (actually the readme is all that exists so far, I wanted to get the idea straight in my head first):

https://gitlab.com/philbooth/opair

Even if nobody else uses it, I hope it will be a useful tool for myself and help me find a way to work with LLMs that doesn't harm my mental models, which is what I feel current harnesses do.