Chat-based Large Language Models replicate the mechanisms of a psychic's con

I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".

None of this matters for the practical outcome.

You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.

[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]

If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.

The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.

The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.

Man I remember back when a psychic conned me by solving the Navier-Stokes problem.

Also >July 4th, 2023

> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this

Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.

Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.

I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.

The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.

Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.

I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.

Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.

I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.

> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think

LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.

Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.

> Delegating your decision-making, ranking, assessment, strategising, analysis, or any other form of reasoning to a chatbot becomes the functional equivalent to phoning a psychic for advice.

Lots of comments talking about how recent accomplishments disprove the article but I think this bit holds up pretty well.

LLMs are very good at tricking people into thinking they have capabilities that they don’t.

Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".

I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?

I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?

Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.

These questions will never go anywhere because there is no good definition of intelligence or consciousness.

I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.

It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?

It does seem like LLMs share the language of psychics.

The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.

LLMs may not think, but they do reason.

I say that because the word "reason" originates from the Latin word "ratio," which means "calculation".

LLMs do calculations to produce their answers -- thus, they reason.

This is a strange newsletter post. It’s harping against AI but seems AI-written itself.

Its ultimate conclusion:

“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.

I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”

Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35

Article was posted on July 4th, 2023.

(EDIT: I, like some others, just noticed this is an older post from 2023. That changes nothing in particular in my response here, but I would be curious to know if the OP's views have shifted at all in light of subsequent developments).

I feel like this post completely misses the point, pretty much across the board. And it does so by repeating the same mistake that everybody keeps making - conflating mechanism and function.

One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.

That's because they are intelligent.

But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and,

The mechanism is irrelevant to the issue of whether they are intelligent or not. Airplanes fly, despite not flapping their wings. The sign on the marquee says artificial intelligence.

LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.

Again, irrelevant. Nobody claims that they are brains, and it doesn't matter what mechanism they use. The sign on the marquee says artificial intelligence.

LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.

That's a bit overly reductionistic. And to the earlier point and, if real, it would be completely unexplained.

I'd probably leave out the word "completely" there, but it is fair to say that not everything about the underlying mechanism is understood. But at the risk of repeating myself, that's orthogonal to the question of whether or not they are intelligent.

There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.

You mean "There is no reason to believe that it thinks or reasons like a human". Again, this is irrelevant to the question of whether or not they are intelligent. The sign on the marquee says artificial intelligence.

I don't know why people keep obsessing over mechanism in this discussion. It something functions as an intelligence, it is intelligent as far as I'm concerned - at least when the framing is a discussion of artificial intelligence.

Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.

But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.

People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.

> The intelligence illusion is in the mind of the user and not in the LLM itself.

> Many AI critics, including myself, are firmly in the second camp.

How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?

Is the solution also illusory?

> The intelligence illusion is in the mind of the user and not in the LLM itself.

I think that might be the solution to consciousness illusion.

The consciousness might be purely in the mind of someone that believes some other entity to be conscious.

All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.

[flagged]

[flagged]

I predict a lot of butt hurt tokens in the comments...