Really using port 22 is very ill advised anyway because you will get constant nuisance brute force attacks (accomplishing nothing because you're using keys or certificates I hope) but still eating up cycles for the crypto handshake.
I've had SSH open on a static v6 that isn't even SLAAC or temporary, it's not my/58::1 but not far off and in DNS, and I have not in 8 years seen a single scan or connection attempt over IPv6 (other than myself). This is not to say there is no risk, but it really is a night and day difference.
Really? I get somewhere in the region of none to barely any, depending on the server.
I mean, yes, you'll get a constant stream of them on IPv4, but why would you run a server on v4 unless you absolutely needed to? The address space is so small you can scan every IP in 5 minutes per port, and if you have my v4 address you can enumerate every single server I'm running just by scanning 65k ports.
Meanwhile, on v6, even the latter of those takes a thousand years. How would people even find the server?
By that same logic, using IPv4 is ill-advised because I could easily give the ssh endpoints their own IPv6 addresses, avoiding the need to hide behind non-standard ports. Scanning through 18446744073709551616 addresses is going to be a lot slower than scanning through 65536 ports.