Almost as if you shouldn't be banning users because of their IP unless that IP specifically has openly attacked you.

Or I guess you can just DENY ALL.