I don't whitelist IPs for ssh anymore, but I always run sshd on randomly selected port, in order to not get noticed by port scanners.
I do it for a really long time already, and until now I am not sure if it has any benefit or it's just umbrella in a sideways storm.
This won't hide you completely, but it will reduce log spam.
My sshd only listens on the VPN interface
As long as you understand it's security by obscurity, rather than by cryptography.
I don't think it's wrong, it's just not the same as eg using a yubikey.