> I SSH into that server with key-forwarding

Almost back to point A then. If the server is compromised in some way, they can use (not copy) OP's keys and use them to clone repos/inject code/etc..