Campaign speech isn't simply the right to address whatever demographics your mail services happens to have decided matter (in the context of a campaign, that is itself a political decision, core protected activity). It's the right to organize around specific voters.

In local politics (which is where I engage mostly), these kinds of decisions get made on individual voter bases at times.

I want to be clear here that while I believe the principles I'm describing to be normatively good, I'm also being descriptive; the restrictions you'd advocate for would almost certainly be held unconstitutional.