It would also be a huge security risk. But I can't think of any fundamental difference with Google queries, other than the sheer entropy of user data involved.

And I'm not a tinfoil internet anarchist, but just because Google only leaks user data in aggregated form to advertisers, doesn't mean that they don't leak their user data, it's just that they did so in a legal and responsible manner.

Maybe considering the difference in data volume and intimacy between queries and AI conversations, the privacy implications of advertising merit a difference in treatment, but I wouldn't be surprised if that is lost to a more simple 'Google did this so we can do it too' momentum.