I know what you mean.

But the numbers don't lie. In my case, I locked down to a fairly small group of European countries and the server went down from about 1500 bot scans per day down to 0.

The tradeoff is just too big to ignore.