And to be fair - that how medicine works today. We don’t really understand what goes into our body, so that’s why we do clinical trials. And sometimes we discover adverse side effects years and decades after a drug has been released.

Agreed. IIRC the mechanism for anesthesia in unknown to this day. In fact, medicine as a whole is very pragmatic and time and time again prefers using techniques with unknown mechanisms but proven results than waiting for a reasonable scientific explanation of what’s going on.

I don't expect Tao to be well versed in medicine, but yeah, that slide is detached from clinical practice.

Medicine has never required that the method of action for a treatment be fully understood. Our current regulatory framework only checks for safety and efficacy because we've never formally understood everything going on in the body. Seems like a difference between "hard" science and the clinical/engineered implementation.

TBF, math is the most ivory-tower of the sciences in that every claim in the field has to be fully understood by proof.

I don't know how to word this in a way that won't get me into a pointless semantics argument, but the word Cure in the slides is clearly used to mean "substance intended to be (but not necessarily confirmed to be) a cure". If you think it's badly worded that's fine, but ostensibly the point here's not to "win" the argument game, but to get his intended point and engage with that.

> This argument is so silly against reality

Ironically, reality people are ok with not understanding everything because they believe another human who understands it has looked into it. It may come a day when we won’t need that but that day is not today. Don’t get me wrong, I’m not arguing about the merits of it, it’s just that at least this is the reality of this timeline on this planet. Not sure about what alternate reality you’re talking about.

> This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.

While the biological effects of any particular chemical still require a great deal of trial and error to determine, drug chemistry itself is unambiguous. Pharma companies employ expert chemists and chemical engineers. They don't manufacture drugs just by randomly mixing together chemicals. At the very least, they must understand what they are making thoroughly enough to determine its physical properties and design a reliable and commercially viable manufacturing process.

Mathematicians inhibit and explore a world that is completely made up and exists only in abstract.

I wonder if this makes them slowly get disconnected from the lived realities of billions of ordinary people.

>This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.

he isnt saying that person who puts stuff into their body should understand it

it is that someone (expert) should understand it to the point he can vouch for it

Has the OP text been changed?

> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?

Now:

> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands the mechanism behind this cure?

I would agree with the second, not necessarily the first.

We don’t even know how acitaminophen works, thousands get their livers destroyed every year from it when alternatives exist, and we are yet to ban it. So how is all of the new reality any different?

I think it is more disconnected in thinking people want to understand the mechanism. As a stage IV cancer patient, I’ve met many people who decide not to continue treatment with approved drugs due to side effects. The notion that every cancer patient will do everything possible to stay alive isn’t true. Most people have a breaking point.

Funny enough, it is a good argument with just a sprinkle of reality for context: Assume effective alternatives to this cocktail exist. Assume you had access to these effective alternatives. Can this be combined with other drugs? Surgery? You say you just need to cure it. You must consider the deeper mechanisms at play to consider all the possible options for living.

He’s making the same argument I’ve heard software developers make for the past 3 years. Lawyers are saying similar types of things. Knowledge work isn’t as special as we all thought it was. Now, it’s Terence Tao’s turn to go through the same motions we’ve had to go through over the past few years.

The problem is you won't be faced with a binary choice of certain death or a single unknown AI cure: You will have a choice between several established protocols AND several unknown AI drugs. You doing a dice roll or do you want to make an educated decision?

Would you take an "AI-designed" airplane where no human understands its physics?

Would you drive on an "AI-designed" bridge unvetted by expert engineers?

Congratulations, you’re patient zero for an unstoppable virus

Slide 22 (from a satirical social media post) explains Tao's opinion, IMO:

> You're absolutely right, I did hack the FDA in order to falsify safety data - that's on me. But here's what's true: 3-methyl-5,5-difluoroazamicazide isn't just ineffective at treating pancreatic cancer, it's lethal.

> t's clear he hasn't spent much time around average people.

You haven't spent too much time around academics then. These people more often than not have not spend a single minute talking to the layman.

It doesn't even need to cure cancer, just prevent it to some unknown degree with unknown side-effects. As long as they want people to take it, they will.

I think his argument is more "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?" So an AI agent could game (lack of a better word) the process and produce something that may cure cancer, but misses something else.

Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.

This is the take I don’t see enough and it’s blisteringly obvious to me and it’s unclear why that it’s not more obvious.

The amount of things people understand about the world is effectively zero, yet they rely on them all the time.

There was a time where people did not get on airplanes because they did not understand them and they did not think that they were safe. Now it’s mundane

I think the answer is that it’s all about the rate of change

Most people cannot understand how things are changing, and even if they don’t understand the changes themselves they do what others around them do and then just build their own model of stability around that.

If the rules of society flip every couple of years then there’s no stable baseline that people can get used to and they all freak because there’s nothing keeping their environment stable

The cancer thing is just a contrived analogy. His real concern is having himself and his mathematician friends replaced by AI. Appealing to someone's health is an easier sell and then it can normalize the relationship of medical priests, and math priests, etc.

> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?

No, and if that was the burden of proof required for medical treatment, every surgery would be done without general anesthesia, because we have no idea how it works.

I would imagine Terence Tao would opt for general anesthesia if he was going to have surgery where it is typically used, so the analogy is obviously flawed.

The problem with his argument is that he put the cart before the horse. He presupposes we've found and validated a cancer cure, but at that point any rational person would accept said cure. The moral quandary is how you are going to validate it against all the other possible candidates, when you don't have a coherent rationale for it to work.

> but it's clear he hasn't spent much time around average people.

Have YOU? It's obvious to me that his example is a rhetorical device which you're taking literally. Do you also realize he's not talking about an actual cocktail? You need to get a diagnosis ASAP.

It's science (understanding) before engineering (efficacy).

The most common Science™ failure mode.

What if it takes your five years to live down to two in horrible agony?