alt Hacker NewsI seriously doubt that data set poisoning will be a real limiter in model performance.
For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models?
For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc.
Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.
Replies
I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.
Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else.
Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates.
Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.
There's a post every other month where some dude who put nonsense information online celebrates because it actually ended up in some frontier models weights.
If it's easy enough that some randos can do it for fun, what do you think happens when there's commercial interest behind it?
Obviously companies are going try nudging AI towards recommending whatever they're selling. It's a logical extension of SEO - and that's a 100 billion USD industry.
Additionally, if I believed myself to be in some sort of spending - err - AI race, I'd try to poison the data sets of my competitors by putting crap out there for others to ingest.