Most people aren't murders either.

If your system collapses under adversarial attacks it is not a resilient system. Bad actors exist and identifying them can be very difficult, especially when you can't instantly violate their rights in many cases.