I don't like this framing.
For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.
Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.
This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.
But I can't build a mobile app with this harness, or make a movie.
These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.
Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.
Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
I am thinking about how to measure the efficiency of ultilzing existing context in one specific domain.
hey! would you be so kind as to share your method or point me in the direction towards resources on how to setup a harness as you describe?
I've been tinkering with harnesses recently but have mostly used github copilot on vscode mostly thus far as my ai tool and feel like the harness setup on there is lacking (or really confusing...). many thanks if so :)
just trying to learn without googling "how to setup a harness" and get bombarded by a trillion yt vids or competing harness providers lol