As a counterexample: I use Matrix along with ~30-50 people, on a federated server, and every room is encrypted. After sufficiently stressing to people that they need to save their secure backup key, we've had few problems with encryption usability.