alt Hacker NewsI work in drug discovery and AI and his slide about cancer medicine isn't very well informed. We have models and narratives about how drugs work, but they are woefully incomplete.
I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically, in real time, using the most advanced mathematical models the world has yet created.
Replies
I think everyone agrees that it's totally fine if a drug came about due to AI.
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
In medicine, or drug discovery more generally, there's FAR too little empirical data for training of ML generally. See OpenAdmet. The kickback to "oh but yeah cancer" is currently just hype. DeepMind has pivoted to this realm but has no demonstrable improvements. For the typical phase 1-2-3 pipeline of drugs, stretching many years, there's not yet any demonstrable improvements.
The effect of the AI-created drug is saving someone's life.
The effect of the AI-created proof is a mathematician abandoning years of research, losing grants, awards, ruining their career, etc.
The only difference here is our emotional reaction!
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.