Do you also federate?
Connect with other matrix servers? As far as I understand, this is the hard part and makes scaling difficult. (One of a servers user joining a group on a different server will make its original server load all that group data as well and that sums up)
For your use case I also considered setting up a instance. Connecting a cluster of friends. But it would be nice if it could grow a little bit, without problems and the server choking.