The exceptions you're listing are mostly real but much more narrow than you're thinking. Only threats of imminent violence are excepted; it's completely protected by the First Amendment to say things like "it's getting to a point where we might have to be violent" or "I would commit violence if suchandsuch were to happen to me", even if other people reasonably find those statements to be stressful and alarming. Similarly, while defamation as such is illegal, it's not defamatory to say something like "John is a stupid fool who can't be trusted", even if I know many people think he's smart and trustworthy and even if his reputation is ruined by my statement.

There's simply no general principle that speech which harms people can be restricted.