I am doing similar with Debian 13 images in QEMU. Only access is to code specifically SSH in. On some, I am taking it a step further and my SSH key is not present at all on the bot. I add it as a git remote from my host into the VM and push and pull from there and then push to GitHub myself. So the agent has no path to GitHub either.
Wow, great notion about using git remote to transfer source from the agent host. I already use ssh into VM to transfer the files like designs for sharing, so going to adopt your approach right now :)