Pre-AI voice recognition (2018), followed by 2 human reviews
https://jamanetwork.com/journals/jamanetworkopen/fullarticle...
=> the error rate was 7.4% in the version generated by speech recognition software, 0.4% after transcriptionist review, and 0.3% in the final version signed by physicians. Among the errors at each stage, 15.8%, 26.9%, and 25.9% involved clinical information, and 5.7%, 8.9%, and 6.4% were clinically significant, respectively.
AI "scribes" in a perfectly replicable best-of-all-worlds scenario (2025): https://bmjdigitalhealth.bmj.com/content/1/1/e000092
=> Omissions dominated error counts (83.8%, p<<0.001), with CAISs varying widely in error frequency and severity, and a median of 1–6 omissions per consultation (depending on CAIS). Although less frequent, hallucinations and factual inaccuracies were more often clinically serious. No tested CAIS produced error-free summaries.
On the gripping hand, people who work in the management end of the US healthcare industry can't be trusted with healthcare or information security to begin with.
Errors can be a significant problem in manual charting as well.
I know a medical professional who does a similar evaluation process to what is outlined in your second link to human written charts. They then use that feedback to guide the department on how to improve their charting.
So, don't presume that those error rates cited in those studies should be compared to a baseline rate of zero. If you review human-written charts, you will often also not have an error rate of zero.
That article is from 8 years ago, accuracy is dramatically better today. We see a few percent error rate.
From the 2025 study: Conclusions The CAISs demonstrate high levels of summarisation accuracy. However, there is great disparity between the currently available CAIS products and, while some perform well, none are perfect. Clinicians should therefore maintain vigilance, particularly checking omitted psychosocial details and medications, and scrutinising plausible-sounding insertions. Purchasers and regulators should be aware of the significant performance disparities identified, reinforcing the need for careful evaluation and selection of CAIS products.
This is exactly what I say and how we teach our people to use it. At the end of the day the human is responsible for the accuracy. We do have providers who decline to use AI because they don't want to double check it, and that's fine by us.
> On the gripping hand, people who work in the management end of the US healthcare industry can't be trusted with healthcare or information security to begin with.
No, this blanket statement is far to overly broad. Health insurers are by far the least trustworthy. Provider organizations are a very, very different group. In my 12 years I have never had a PHI breach or leak that wasn't a human making a mistake. No hacks, no credential breaches, no backdoors or zero days, no network infrastructure penetrations. Two former employers had breaches years after I left which I think speaks well to my track record. I take security incredibly seriously. Our patients are the most important part of my job.
> On the gripping hand,
It’s been a year or so since I last read The Mote In Gods Eye/The Gripping Hand but I randomly was thinking of this morning. Very funny that I would see a reference to it the same day.
My dad likes to joke around and his doctor uses some kind of transcription service. Time for fun!
His doctor asked him about using drugs and he made a joke that was something like "I only use coke" - meaning coca-cola. Of course his doctor knew he was kidding about drinking too much soda because he eats/drinks too much sugar. So they had a little laugh and moved on.
BUT now it's in his medical transcripts. My mom said it "transcribed" it as something like "the patient responded he has used cocaine recently".
I guess his doctor doesn't go in and actually fix things or even read over what the transcription says...
Also both of my parents have accents and have reported really weird transcriptions that don't match what they actually said.
So now my mom has told my dad he can't make jokes with the doctor anymore because even if the doctor knows he's joking it's going to get noted down as a "fact".