That's how they get you. They are trained to make you like them. They are not trained to convey true information.

I find them irritating because they will link a source entirely unrelated to (or worse, directly contradicting) the fact they are claiming. The Google AI summaries might be the most blatantly hallucinating model widely deployed right now.

I’ve had them abruptly try to turn mathematical theorems into cooking recipes before cutting off mid-sentence in a panic.

YMMV, but nearly every one I read has a glaringly incorrect comment. I frequently research off-trail mountaineering routes and the incorrect bullshit it confidently spouts at the top of a list of results (results which very much counter the AI overview) is liable to end up getting someone hurt, lost, or worse.

Not every technology needs to exist simply because it’s possible.