That's why you need a server. e2ee does not help with any of that.