alt Hacker NewsThis article is about measuring how often an AI missed cancer by giving it data only where we know there was cancer.
> Am I missing something?
Yes. The article is not about AI performance vs human performance.
> Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern
Ironic
The article has the headline "AI Misses Nearly One-Third of Breast Cancers, Study Finds".
It also has the following quotes:
1. "The results were striking: 127 cancers, 30.7% of all cases, were missed by the AI system"
2. "However, the researchers also tested a potential solution. Two radiologists reviewed only the diffusion-weighted imaging"
3. "Their findings offered reassurance: DWI alone identified the majority of cancers the AI had overlooked, detecting 83.5% of missed lesions for one radiologist and 79.5% for the other. The readers showed substantial agreement in their interpretations, suggesting the method is both reliable and reproducible."
So, if you are saying that the article is "not about AI performance vs human performance", that's not correct.
The article very clearly makes claims about the performance of AI vs the performance of doctors.
The study doesn't have the ability to state anything about the performance of doctors vs the performance of AI, because of the issues I mentioned. That was my point.
But the study can't state anything about the sensitivity of AI either because it doesn't compare the sensitivity of AI based mammography (XRay) analysis with that of human reviewed mammography. Instead it compares AI based mammography vs human based DWI when the humans knew the results were all true positives. It's both a different task ("diagnose" vs "find a pattern to verify an existing diagnosis") and different data (XRay vs MRI).
So, I don't think the claims from the article are valid in any way. And the study seems very flawed.
Also, attempting to measure sensitivity without also measuring specificity seems doubly flawed, because there are very big tradeoffs between the two.
Increasing sensitivity while also decreasing specificity can lead to unnecessary amputations. That's a very high cost. Also, apparently studies have show that high false positive rates for breast cancer can lead to increased cancer risks because they deter future screening.
Given that I don't have access to the actual study, I have to assume I am missing something. But I don't think it's what you think I'm missing.