I keep hearing that, and I have yet to go there. I find the permission checks are helpful – they keep me in the loop which helps me intervene when the LLM is wasting time on pointless searches, or going about the implementation wrong. What am I missing?

If you're gonna do that, make sure you're sandboxing it with something like https://github.com/kstenerud/yoloai or eventually you'll have a bad time!