You have freedom of speech to advocate for your politics. The rest of us have the freedom of association to not want to be involved with you in any way.

These are not contradictory - they are both essential freedoms.