No-one is "speaking out against" the one who is pointing out any wider discrimination, beyond whatever aribtrary circle you choose to draw.

What is being "spoken out against" is the idea that taking the moral (or political) action within whatever circle you feel able or willing to support is insufficient, or even discriminatory in itself. After all, this is exactly how this conversation started. Good for you if you want to change the world - let's not forget 3rd party discrimination against other 3rd parties! For many of us, it's one of numerous pressing problems to be addressed. If you wish to bring privilege into it, having the freedom to make fighting any and all discrimination a primary concern is a sign of privilege that few have.