I have been putting my agents on their own, restricted OS-level user accounts for a while. It works really well for everything I do.

Admittedly, there’s a little more friction and agent confusion sometimes with this setup, but it’s worth the benefit of having zero worries about permissions and security.