it's not the mere lack of being with, it's the enthusiastic, public advertisement that you are not with. If you are actually neither with nor against, then you don't have any need to say anything about the issue at all. If you make a big deal about not being with or against, most people who do that are against but don't want to say so.