I think the most secure setup, though not so convenient for the average user, is a separate machine with QEMU/KVM. The machine should be isolated adequately, such that even if compromised, it shouldn't be able to cause damage or gain access to other machines. Additionally, a proxy server on your machine or elsewhere could hide sensitive credentials. A helper binary on your computer would then control spawning new disposable VMs with premade images and your SSH key. The images can be lightweight or the desktop version with a remote VNC.
I agree with this, virtual machines are invented to solve the sandboxing/multi-tenant issues.
This is why ec2 and the likes all sell you access to virtual machines (dividing up their underlying hardware).
I use vagrant on a seperate machine in a seperate network. The magic of ssh makes it transparent for me, and I feel pretty sure the agents cannot get to stuff that matters.
quickemu[0] is sort of amazing for these type of use cases actually. I feel like I could use it more for this type of stuff.
Gondolin[1] is what you are describing. It's made by the same person who made the Pi coding agent and sends all of the agent's bash into a small QEMU vm.
[1](https://earendil-works.github.io/gondolin/