Touche.. actually a good point, but actually those are two different situations. With one, I'm accessing a website and trusting that the certificate is signed by someone I trust; so the trust in my browser certificates (which include certificates from hundreds of certificate authorities all over the world, any one of which could be compromised, robbed, or controlled by an adversarial person or even government) is extended to the site that I'm visiting. To say this is weak sauce rather understates how bad this actually is. (To paraphrase Churchill, this is the worst possible design, except for all the rest.)
With the other, I'm logging into a server for the first time (and I could simply deploy the same trusted host key to all my ssh servers via an autoscaling configuration or whatever). I think it's debatable if TOFU is worse or better than your (granted clever) metaphor.
(to those who'd recommend userify, yes - great for the client login issue and definitely increases security, but to parent's point, TOFU is still needed unless you want to distribute host pubkeys)