It’s a classic example of the base rate fallacy. The judge sees that a system with a seemingly high accuracy rate (like 99.999% accurate) has flagged a person, and they assume that means the person is highly likely to be guilty.
However, the system uses a dragnet approach, and is checking against millions of people. If you are checking 300 million people, that 99.999% accuracy check is going to find 3,000 people, and AT LEAST 99.96% of those people are going to be innocent.
This is why we can’t have wide, automated surveillance.
It’s a classic example of the base rate fallacy. The judge sees that a system with a seemingly high accuracy rate (like 99.999% accurate) has flagged a person, and they assume that means the person is highly likely to be guilty.
However, the system uses a dragnet approach, and is checking against millions of people. If you are checking 300 million people, that 99.999% accuracy check is going to find 3,000 people, and AT LEAST 99.96% of those people are going to be innocent.
This is why we can’t have wide, automated surveillance.