I think a self-hosting option for IM that's better than XMPP could be nice... that said, not sure if something like torrent would be good. I gave a lot of thought to something similar for emails, and the biggest issue came down to, it would be difficult to do something anonymous, distributed, and resistant to flooding/poison pill attacks.

Torrents themselves work against this, because you have known hash values as part of seeding... with email and messaging, you wouldn't have one-off advanced knowledge, and if anyone can send anyone a message, you'd be open to a flood of messages from what seems to be randos. There's some of this from scammers on Telegram and other social media, but it would be much worse.

A federated system that's otherwise tethered to a domain/email or similar would at least allow for self-management and/or block listing techniques to work better in practice.

There were some rather forgotten nowadays projects like Tox, Briar, Ricochet that should fit what you describe