You're finding comforting explanations to allow you to act dismissively towards other people. I understand this is a strategy that is popular these days, but maybe consider how another fellow human will feel when you "don't give them a reason why you stop interacting, you just leave", and judge what they tell you as "toxic waste"; and how you might be the one to make it worse for them (and yourself). If you mentalize yourself into the other position, yours might appear arrogant and condecending if not psychologically violent from where you stand ("how's your aura looking?").

If you feel worn out after listening to other people, that's one way to avoid that, at the expense of human connection. There are other ways to not feel drained even after listening to the most horrible (or boring) stories that don't cut people (and thus yourself) off. You gain options, not lose any. You can learn to have more control over your own inner state without effort, and become more independent from what people around you are saying or doing, instead of turning your problem into their wrongdoing. Instead of having your world suddenly be full of energy vampires you need to protect yourself from.