See the agent as a coworker ssh-ing on your machine, how would you work efficiently ? By working on the same directory ? No

You give each agent a git worktree and if you want to check, you checkout their branch.