It's so slow because it's so badly designed as a protocol, E2E isn't really the problem (the slowness is roughly equivalent for non-encrypted rooms)