What? Freedom of association implicitly means freedom not to associate. It is not at all incompatible with freedom to say, "I don't want to hang out with those guys because they suck."

I believe in freedom of speech for people that I don't want to talk to. There is no contradiction in that.