People proudly voting for parties and policies that demonise trans people, of which I know many. I cannot be your friend in good conscience if you're willing to destroy the lives of my other friends.
How are their lives being destroyed?
Being told that you have to follow the same rules as everyone else for e.g. spaces designated to be used solely by the opposite sex, doesn't seem so bad.
That is, by definition, indirect. So that doesn't qualify as "directly harming" anyone, even if your analysis of those policies is otherwise accurate.