I find that the "what if a cabal of small server operators defeds your server" risks very frequently overblown, although it is good to be aware of it.

As a correction, though - emails get bounced based on opaque reputation rules (domain-related or otherwise) all the time. Email and fedi are very similar in this respect.