alt Hacker NewsI have a distinct line between when I'm willing to believe an LLM's output and when I'm not: whether I would believe the same thing from an anonymous Internet forum post or a blogger I don't know. Those posts are not unlikely to be misinformed, biased, lies, or otherwise untrustworthy. And yet, I spent plenty of years honing a sense of when they were good enough for certain things.
Replies
One of the things I do semi-frequently is look for the evidence that some concert took place 15+ years ago. Or maybe I already definitively know it happened, but not exactly at which venue or the exact date of the concert. This I feel like is a non-trivial task, but one with a very definitive answer whose evidence more often than not still exists somewhere online.
In my experience every LLM out there is utterly useless and quickly defaults into "here are other concerts that took place around that time near that location". Google Search (ignoring the AI overview) is even more useless, as it refuses to show literally any webpage that's older than say 5 years. YouTube search is genuinely better than Google at surfacing old and grainy fan-made videos uploaded in like 2010, but also defaults into synonyms nonsense pretty quickly.
But, the search functionality of exactly one forum and three local news websites that I know have an archive that dates back long enough beats every single one of those abovementioned every single time. Three people are talking about their experience at a concert on a random 15+ year old forum thread? It happened. The tiny list of 5 or so (Google-hosted!) Blogspot blogs I have bookmarked? They usually have a photo of the ticket that Google Images refuses to show me.
Not only are search engines completely dead as a category, but LLMs are a shit replacement for them. "We" (okay, Google specifically) has truly committed a crime comparable to burning the Library of Alexandria. Everything older than a decade that wasn't properly documented on Wikipedia is just gone, never to be seen again.
Yeah, it's like everyone was under the impression you could just trust the internet before LLMs.
It's a great tool, but verify the important things (or do them yourself)
I'm not sure if this is your take, but it feels an aweful lot like an AI "Good enough is good enough" handwave.
An anonymous answer to a question is more trustworthy to me. They have no reason to lie. They are usually answering out of kindness. At least they used to be. Now it’s often actually bots advertising a product or pushing something, pretending to be a helpful user with an anecdote and a good experience using a niche product.
AI shouldn’t have any reason to lie. But its lies aren’t intentional. It’s just actually making things up and “hallucinating” when it pretends that an option or setting exists, or confidently claims something entirely untrue, and makes up a source to go with it. For something Google is willing to shove into the top of every search result it’s crazy the percentage of time the answer is blatantly incorrect.
A lot of that sense was probably based on side channels like proper grammar, writing style, etc. That’s all gone now :(