I am not sure the supervision is the problem, it's more the way Claude is acting in general, taking initiative that could burn thousands of token in a second. Just today I ask it a small fix on a newly pulled project, it wrote 5 lines of code but then immediately started to run the full test suites, which itself runs in docker, with several images that I didn't pull yet so it started to download everything and by the time I click stop, it already filled the context windows with dozen of hundreds of useless lines... I find it very hard to fins the sweet spot in the prompt between "do not assume, do not take initiatives" and "inform me if there is a simpler way to do what I am asking or if there are some downsides I may not have thought about".

> I think running a model unsupervised in anything above medium is a sucker move to burn more tokens.

What about errors compounding and any on-the-go decisions being made being the wrong ones? That kills anything long form, no matter how good and detailed your initial plan is - there will always be something along the way.

The models would need a way to identify a difficult problem and apply max reasoning there themselves and cruise through everything else at a lower level.

That’s my experience too, going off the deep end with CoT doesn’t meaningfully improve output on a lot of tasks but it does incinerate tokens and take longer

My favorite part (codex thing) is how it runs all commands twice, every time.

Once on a sandbox, realizes nothing works in its sandbox, then again outside the sandbox.

Fucking hell.