alt Hacker NewsThe point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cancer.
In nearly half the false negatives from both the mammogram and DWI datasets, the cancer was categorized as occult by two breast radiologists, meaning the cancer was invisible to a trained eye. The AI model's non-occult false negative rate on the mammography data is 19.3%.
For that 19.3% figure, see Table 2: 68 non-occult in AI-missed cancer, 285 non-occult in AI-detected cancer.
This study did not compare the AI to a radiologist on a mixed set of healthy and cancer images.
Replies
What sensitivity / specificity are trained radiologists able to receive?
I'm not a medical researcher, but I am a computer guy; I was struck by something very different in the papers - the abstracts at least refer to "AI CAD" as what they're testing - no software information, no versioning - on the CS side, this stuff is of paramount importance to make sure we know how the software performs.
On the medical side, we need statistically significant tests that physicians can know and rely on - this paper was likely obsolete when it was published, depending on what "AI CAD" means in practice.
I think this impedance mismatch between disciplines is pretty interesting; any thoughts from someone who understands the med side better?
Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.
As it was retrospective study, I really hope they made sure that test images were not in a training set of the algorithm. If they were, the whole study is meaningless.