> kinda worries me (given also IP spoofing is possible?),

It's not, Tailscale authenticates incoming connections. (Note that we're not talking a regular SSH connection to the server's public IP here. You'd connect to the server's SSH daemon through Tailscale.)