Ignoring that you've just cut off a whole vector of usefulness, how do I keep it from exfilling my inbox to the Internet in response to a malicious email? Or using its access to take control of my online accounts?

Honest question, this kind of stuff is what keeps me from using it.