This won't hide you completely, but it will reduce log spam.

My sshd only listens on the VPN interface