I think part of the “just use a CLI” crowd might also be building similar agents as ChatGPT and Claude.ai web interface. I know at least 4 teams doing that in one company.

All those teams, including ChatGPT and Claude.ai, have figured out that you will eventually need to give your agent a small sandbox Linux environment to unlock the same level of “intelligence“ those coding harness exhibit. Stitching together the results of a cli command through scripting or coding gives the agent a ton more flexibility in what it can do as it can utilize its text generation capability into executable logic. toolcalls mostly work for actions rather than complex and novel problem solving. You are making the agent represent a programming control flow through toolcalls while carrying the context between them in a lossy, nondeterministic, wasteful, slow and rigid way.

It’s one thing if you want to artificially limit that agent to a very strict set of available APIs that it must use in a specific way while transferring context between them through the LLM and you don’t want to incur the cost of the extra sandbox compute. But coding harnesses have demonstrated that letting the agent write a small shell or python script can let the agents solve problems that you haven’t even really anticipated in your toolcall approach or that tool calls make prohibitively expensive or not even possible.

But also the token cost tends to dwarf the sandbox compute cost, so why not pay the $0.05/hour to have a sandbox where the agent can run free when you are already paying orders of magnitude more for the tokens