I ended up using guix shells (container mode) for my agents and sharing just the directories they need