alt Hacker NewsChatGPT Health is a marketplace, guess who is the product?
Comments
Fascinating article.
Data security will be another important factor in whether we should choose our private health information with these third parties or not.
Manage My Health in NZ was hacked earlier this week: https://www.rnz.co.nz/news/national/583417/who-are-the-hacke...
A phrase I liked to describe what we're doing with LLMs is "building a personal panopticon". The benefits are immense but you're placing a huge bet on the landlord of the tower.
Google did that, Facebook did that and every other company who boasted their user-base numbers did that. They sold user attention and harvested user data. Nothing new here.
Do users find value in it? Thats the ultimate question. I think it is a resounding yes.
If it is genuinely beneficial, this will become an open source project that everyone is able to run with a local agent in their house that runs cold. I will make one if no one else will, but discovering how to make it ubiquitously helpful and not drought with legal liability is challenging. I welcome a company willing to take this early risk.
ChatGPT mostly refuses to talk health issues , while i have found Gemini is reasonably cooperative when asking for things like symptoms and treatments .
This makes me not wanting to try out their new offering.
LLMs for medical info are good, but they're easily abuseable. I've got a friend who is an anxious mom. They use gpt/Gemini to "confirm" all of their suspicions and justify far more doctor/medical visits than is at all reasonable, while also getting access to more recurring antibiotics than is reasonable. LLMs are basically giving them the gun powder to waste the doctor's time and slam an already stressed medical system when all their kids need most of the time is some rest and soup.
Understand what you are trying to say but without giving an alternate solution what the reader would do with your thoughts?
The HHS is asking for recommendations on how to leverage AI for healthcare: https://www.hhs.gov/press-room/hhs-ai-rfi.html
This is probably part of an effort to position them a potential vendor to help the government with this.
This sounds like it will be one of those products which starts out as an optional service, but eventually becomes required to use if you want to participate in the overall healthcare system.
I think the consequences of hackers obtaining health data like this would be unimaginable. OpenAI is far inferior to Apple when it comes to privacy and security protection.
Not mentioned in the article, but one interesting area where OpenAI could play is in participant identification and recruitment for clinical trials. In fact, ChatGPT could also help operate the clinical trials which is a highly paperwork intensive business, and therefore something that AI could add value to.
Ultimately pharmaceutical companies pay up to $100,000 per participant to hospital networks these charges must be itemized as expenses from the hospital on the most part (bounties are illegal usually.) open AI would provide a cheap way in for pharmaceutical companies to identify participants given that OpenAI has an incredible perspective into the physical and psychological state of their users. Imagine how much more is shared with OpenAI compared to a clinical trial coordinator at a hospital when a psychiatric drug is being tested.
This would also give OpenAI leverage in partnering with pharmaceutical companies. OpenAI executives have stated this is a goal, but otherwise they’ve made little progress on it.
It’s wild to imagine - someone with borderline personality disorder having delusional conversations with an AI chat Bot for six months, receiving an offer to participate in a clinical trial, and then having their subsequent AI conversations used as evidence to analyze the efficacy of the drug. The ironic thing is if that person had delusions about hidden forces listening to them…they’d be RIGHT!
Me and I hope they are selling me something to fix my RLS
I would give a lot of money to do so
I find it ironic that the article is warning against AI use while it uses an AI-made cover image. Surely they find the same fault with copyright issues and AI art? Right?
My level of trust for data:
1) Claude
2) OpenAI
3) Grok
4) Gemini
Dystopian and frankly, gross. Its amazing to me that so many people are willing to give up control over their lives and in this case, their bodies, for the smallest inkling of ease.
The only thing you have control of in this world is your body (men only, women have already been denied body autonomy in the US), so giving this to the very entities that "do harm" as opposed to those who pledge to "do-no-harm", is straight up bonkers.
It's not the data or the use of said data for the intended purpose. There is a law of sorts in life that says what ever they promise, it will be broken. The data and its intended purpose will be perverted and ultimately used as a weapon against the very people who provided the data.
There’s no real smoking gun here showing what they are taking, storing, and using (or how they’re using it). But I do agree that nobody should dive in to this until that’s better understood as it is likely they are not following HIPAA and will not respect your privacy. Definitely not when there’s money on the table. Given their track record the concern is warranted.
ChatGPT has made a material difference in my ability to understand health problems, test results, and to communicate with doctors effectively. My wife and I were talking last night about how helpful it was in 2025. I hope that it continues to be good at this.
I want regulators to keep an eye on this and make smart laws. I don't want it to go away, as its value is massive in my life.
(One example, if you are curious: I've been doing rehab for a back injury for about 10 years. I worked with a certified trainer/rehab professional for many years and built a program to keep me as pain-free as possible. I rebuilt the entire thing with ChatGPT/Gemini about 6 weeks ago, and I've had less pain than at any other point in my life. I spent at least 12 hours working with AI to test and research every exercise, and I've got some knowledge to help guide me, but I was amazed by how far it has come in 12 months. I ran the results by a trainer to double-check it was well thought out.)