If our standard for laws would be that "well no reasonable person would do this/believe this" then nothing would be illegal, there'd be no need to label any product as potentiality harmful, etc.

Do you really want to go there? That everything in the world would have a literal "caveat emptor" attached to it?

The harm was not done to the readers of the AI generated response, but to the defamed companies.

And yes, it is ok to remove choice if the existence of that choice violates other person’s rights.

Google can continue offering that choice if they make sure nobody is defamed.

So we deploy a technology no one should trust to the general public, for what exact reason?

Do you mean the responsible people who will ensure their algorithms can be trusted with the important task of acting in the best interest of said people? Try and get a defamatory statement about google from the AI search box.

I have to fight with my family members when they "Google" something, read the top AI slop result, and I ask which page it came from. They believe what is on the Google landing page, and actually I don't think that is a naive assumption. Google has pushed itself as the information oracle, now they are delivering slop as the first result. It's a bait and switch.