I think everyone agrees that it's totally fine if a drug came about due to AI.

I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.

To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."

* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)