While this is true as a platitude, I do find that medical colleagues seem to treat this platitude rather overzealously at times, often completely disregarding the posterior probabilities involved. I like to call this the prior-over-posterior fallacy/bias (though one could argue this is simply a variant of the normalcy bias)

E.g., I get the whole "if you hear galloping think horses not zebras" adage, but I guarantee you, if someone comes and says "hey when I was in Africa I saw a black and white striped animal galloping really loud", I bet you an uncomfortably large percentage of the "horses not zebras" crowd would still favour a horse over a zebra diagnosis, despite the overwhelming posterior.

Combine this with our (otherwise reasonable) tendency of avoiding the information bias of seemingly unnecesary tests, and it becomes a big problem, whereby uncommon conditions are treated as common, thereby often escaping detection altogether, and driving down their apparent prior probability even lower, causing a diagnostic vicious cycle.