This is not a perfect analogy: when I choose not to join a club, I have a reasonable expectation of that club not declaring me its member, and not fighting me for failing to follow the rules that the club established.

You do if they're bullying you for not joining.