Which the AI then passes on to a user who doesn't know any better and then you get recipes for mixing bleach and ammonia for curing asthma (extreme example).
It's entirely the responsibility of ai companies how much of the internet's trash they deliver to their users
Absolutely.
And for this the company responsible for the AI disseminating this information in such a way that people are relying on it should be prosecuted and/or sued into oblivion.
The original poster of the bad information should perhaps as well, but only also if it was reasonable to rely on them for this.
(We have the legal principles, we just need to use them.)
If I write a story that includes a character mixing bleach and ammonia for curing asthma and publish it on my website, am I responsible for an LLM company ingesting my story without contact or my consent and then at some point in the future providing a user instructions to mix bleach and ammonia for curing asthma?