Agree, this feels like an XY problem.

The real issue is the level of access and capabilities you grant the agent, not where the inference runs or whether it's "sandboxed".