The point of E2EE is that you don't need to trust the server that you're talking through. A secure E2EE scheme is secure in the presence of an actively malicious server.

I think SGX is a red herring in this context. You can find it risky or not, but it's (1) not material to Signal's E2EE property, and (2) materially better than email, which sends all contact metadata in plaintext, regardless of whether you try and encrypt your message bodies. Bringing it up in this context seems like a category error.