> This is insane.

As a human, this is fantastic.

> How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.

Personally, I hope it can't, or the fix is onerous and kneecaps a lot of the utility of AIs.

I wonder if we can even have some adversarial competition, where competing businesses try to get AIs to ingest bad data to pollute their competitors' decision-making.

For starters: lets get some fake math papers out there. Given proof-checking systems, maybe they won't totally stop OpenAI, but hopefully the bad math could at least drive up the cost and slow AI down.

I've been thinking for awhile it is very strange that claude/ openai etc google "What was the inflation rate last quater" instead of just having a tool to query BLS datasets and that sort of thing.

So hopefully most of these statistics move towards first party sources.

I don't think it is up to us at this point. LLMs are now the lens ("filter for what people see"). I hope, they will fix this. I think that they will still rely on reasoning and better grounding, which should prevent straight lies. But I think two bigger problems here: 1. training data might have the BS which would steer reasoning; 2. grounding might still be limited by small number of sources.

It cannot be fixed. The most hilarious example of it is YouTube. They have a video summarisation feature, which automatically creates video summaries that then get fed into the search engine. The problem with those is that they are created using models that are good few months behind, and let's say someone published a video on Nikon ZR and the model used to create summaries does not know about Nikon ZR and inserts references to Nikon Z8 or Nikon Zf into the summaries. This decreases accuracy of YouTube search and negatively affect discoverability. It's the AI version of shitting into the well you drink from. Bon appetit!

>How can this be fixed?

People stop trusting what they read on the internet.

All alternates are just attempting to improve the baseline trustworthiness of random information from the internet (and/or LLM that's baby-birding it back to you). Any increase in how much people blindly trust random stuff from the internet only serves to increase the blast radius when it is wrong.

The fix that seems obvious to me is to only let the LLM see truthful (or at least non-malicious) data. The model trainers carefully curate the training data, and you don't give the LLM access to full-web search. Instead, you hook them up to an 'official' source like an encyclopedia, documentation for the software in question, etc.

It's a ton of effort and expense, but probably feasible from a technical point of view. So instead, the model providers will probably do something silly like wire up rules like "NYT=Reliable;my.blog.forbes.com=Unreliable" and they'll say the issue is solved.

>How can this be fixed?

Defer to authority and trust your government to rank what is truth and what is not, so your LLM always gives you the right answer. Nothing can go wrong here ever.