alt Hacker NewsMeanwhile they are pushing AI transcription and note taking solutions hard.
Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off.
The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia.
I spoke to a practice nurse a few days ago to discuss this.
She said she didn’t think patients would care if they knew the data would be shipped off site. She said people’s problems are not that confidential and their heath data is probably online anyway so who cares.
Replies
The wilful ignorance and total apathy is appalling.
I've had similar experiences in Australia. I emailed one of my docs' practices asking if they use Heidi AI (or anything similar) and that I do not consent. They were using it without my consent.
In the consultation, he tried to give me the schpiel, including the 'it stays local' thing. The Heidi AI website has the scripts for clinicians; he ran through them all.
Oh, their documents for clinicians also mention every two sentences that patient/client consent is not required at all. I wonder why they keep saying that? Hmm.
This doctor knows I am a developer. When I asked him to explain what he meant by 'local data', he said the servers were in Australia. I almost flipped the desk. Aside from the fact that it is mandatory (it's the law! they do not have a choice!), it's ...kind of meaningless where the servers are, especially when he (on behalf of Heidi AI) was trying to sell it as a security or privacy feature. When I pointed that out, he just couldn't wrap his head around it. Of course he can't, he doesn't understand.
AHPRA's "Meeting your professional obligations when using Artificial Intelligence in healthcare" guideline[0] (not any kind of enforceable requirement, unfortunately) has great stuff in it. It encourages using it with the informed consent of patients. Even if my doctor read it and agreed with it, and cared about getting consent, how the hell can he inform patients sufficiently when he has absolutely no idea about, well, anything?
He keeps pushing it and asking me about whether I've changed my mind about allowing him to use it. No! He keeps asking me questions that only confirm he hasn't even done a perfunctory web search about why some people hate LLMs, especially in the context of PII and PHI.
I really do feel for clinicians, but these products are not the answer.
[0] https://www.ahpra.gov.au/Resources/Artificial-Intelligence-i...
I spec'd up an implementation of this that uses a hardware button with colors that is in reach of either party. The customer went with a different vendor based on price/"complexity"/training.
The New Zealand Chief Digital Officer allowed Australian cloud providers to be used as there weren't suitable NZ data centers and this was many years ago.
Didn't Health NZ just suffer a major data breach and have patient records ransomed?
I genuinely think people “care” about it (in quotes). It’s one of the things where nobody cares unless something bad happens, and when bad thing happens, they shrug it off and forget about it a week later.
I’d go as far as saying she’s right. And we’re in a tiny minority for even thinking about it.
It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work for me. And if my colleague ever gets a college complaint I have no clue how he's gonna navigate any AI generated errors. I'm all for AI and it's great for things like copywriting, brainstorming and code generation. But from what I'm seeing, it's creating a lot more headache in the clinical setting.
If you're why doesn't this guy just check the AI scribe notes? Well, probably because with the amount of detail it writes, he'd be better off writing a quick soap note.