ssh-agent will also be happy to provide the key to git after an initial unlock with the passphrase.