You can use agent forwarding (ssh -A) to make your local hardware-backed key available on the VPS.

If you're using tmux on the VPS this can get a little tricky, because by default your tmux panes will inherit your initial SSH client's SSH_AUTH_SOCK; when you re-attach tmux in a different SSH session you'll no longer be able to authenticate with the local YubiKey. But I wrote a shell script that hooks into tmux to keep SSH_AUTH_SOCK working correctly with this kind of setup: https://github.com/mshroyer/socklink