By that same logic, using IPv4 is ill-advised because I could easily give the ssh endpoints their own IPv6 addresses, avoiding the need to hide behind non-standard ports. Scanning through 18446744073709551616 addresses is going to be a lot slower than scanning through 65536 ports.
You don't put your server IP in your DNS? You type the IPv6 address every time?
A lot of servers expose something public so they can be found. Otherwise what's the point of being publicly accessible?