alt Hacker NewsThe Case That A.I. Is Thinking
Comments
Personal take: LLMs are probably part of the answer (to AGI?) but are hugely handicapped by their current architecture: the only time that long-term memories are formed is during training, and everything after that (once they're being interacted with) sits only in their context window, which is the equivalent of fungible, fallible, lossy short-term memory. [0] I suspect that many things they currently struggle with can be traced back to this.
Overcome this fundamental limitation and we'll have created introspection and self-learning. However, it's hard to predict whether this will allow them to make novel, intuitive leaps of discovery?
[0] It's an imperfect analogy, but we're expecting perfection from creations which are similarly handicapped as Leonard Shelby in the film Memento.
Well, I think because we know how the code is written, in the sense that humans quite literally wrote the code for it - it's definitely not thinking, and it is literally doing what we asked, based on the data we gave it. It is specifically executing code we thought of. The output of course, we had no flying idea it would work this well.
But it is not sentient. It has no idea of a self or anything like that. If it makes people believe that it does, it is because we have written so much lore about it in the training data.
This is merely a debate about what it means to "think." We didn't really previously need to disambiguate thinking / intelligence / consciousness / sentience / ego / identity / etc.
Now, we do. Partly because of this we don't have really well defined ways to define these terms and think about. Can a handheld calculator think? Certainly, depending on how we define "think."
Until we have a testable, falsifiable thesis of how consciousness forms in meat, it is rash to exclude that consciousness could arise from linear algebra. Our study of the brain has revealed an enormous amount about how our anatomy processes information, but nothing of substance on the relationship between matter and consciousness. The software and data of an operating LLM is not purely abstract, it has a physical embodiment as circuits and electrons. Until we understand how matter is connected to consciousness, we also cannot know whether the arrangements and movements of electrons meet the criteria for forming consciousness.
This reads like 2022 hype. It's like people stil do not understand that there's a correlation between exaggerating AI's alleged world-threatening capabilities and AI companies' market share value – and guess who's doing the hyping.
all this "AI IS THINKING/CONSCIOUS/WHATEVER" but nobody seems worried of that implication that, if that is even remotely true, we are creating a new slave market. This either implies that these people don't actually believes any of this boostering rhetoric and are just cynically trying to cash in or that the technical milieu is in a profoundly disturbing place ethically.
To be clear, I don't believe that current AI tech is ever going to be conscious or win a nobel prize or whatever, but if we follow the logical conclusions to this fanciful rhetoric, the outlook is bleak.
I've shared this on YN before but I'm a big fan of this piece by Kenneth Taylor (well, an essay pieced together from his lectures).
The Robots Are Coming
https://www.bostonreview.net/articles/kenneth-taylor-robots-...
"However exactly you divide up the AI landscape, it is important to distinguish what I call AI-as-engineering from what I call AI-as-cognitive-science. AI-as-engineering isn’t particularly concerned with mimicking the precise way in which the human mind-brain does distinctively human things. The strategy of engineering machines that do things that are in some sense intelligent, even if they do what they do in their own way, is a perfectly fine way to pursue artificial intelligence. AI-as-cognitive science, on the other hand, takes as its primary goal that of understanding and perhaps reverse engineering the human mind.
[...]
One reason for my own skepticism is the fact that in recent years the AI landscape has come to be progressively more dominated by AI of the newfangled 'deep learning' variety [...] But if it’s really AI-as-cognitive science that you are interested in, it’s important not to lose sight of the fact that it may take a bit more than our cool new deep learning hammer to build a humanlike mind.
[...]
If I am right that there are many mysteries about the human mind that currently dominant approaches to AI are ill-equipped to help us solve, then to the extent that such approaches continue to dominate AI into the future, we are very unlikely to be inundated anytime soon with a race of thinking robots—at least not if we mean by “thinking” that peculiar thing that we humans do, done in precisely the way that we humans do it."
People have a very poor conception of what is easy to find on the internet. The author is impressed by the story about Chat GPT telling his friend how to enable the sprinkler system for his kids. But I decided to try just googling it — “how do i start up a children's park sprinkler system that is shut off” — and got a Youtube video that shows the same thing, plus a lot of posts with step by step directions. No AI needed. Certainly no evidence of advanced thinking.
The author searches for a midpoint between "AIs are useless and do not actually think" and "AIs think like humans," but to me it seems almost trivially true that both are possible.
What I mean by that is that I think there is a good chance that LLMs are similar to a subsystem of human thinking. They are great at pattern recognition and prediction, which is a huge part of cognition. What they are not is conscious, or possessed of subjective experience in any measurable way.
LLMs are like the part of your brain that sees something and maps it into a concept for you. I recently watched a video on the creation of AlexNet [0], one of the first wildly successful image-processing models. One of the impressive things about it is how it moves up the hierarchy from very basic patterns in images to more abstract ones (e. g. these two images' pixels might not be at all the same, but they both eventually map to a pattern for 'elephant').
It's perfectly reasonable to imagine that our brains do something similar. You see a cat, in some context, and your brain maps it to the concept of 'cat', so you know, 'that's a cat'. What's missing is a) self-motivated, goal-directed action based on that knowledge, and b) a broader context for the world where these concepts not only map to each other, but feed into a sense of self and world and its distinctions whereby one can say: "I am here, and looking at a cat."
It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical. I think LLMs represent a huge leap in technology which is simultaneously cooler than anyone would have imagined a decade ago, and less impressive than pretty much everyone wants you to believe when it comes to how much money we should pour into the companies that make them.
I've written a full response to Somers' piece: The Case That A.I. Is Thinking: What The New Yorker Missed: https://emusings.substack.com/p/the-case-that-ai-is-thinking...
The core argument: When you apply the same techniques (transformers, gradient descent, next-token prediction) to domains other than language, they fail to produce anything resembling "understanding." Vision had a 50+ year head start but LLMs leapfrogged it in 3 years. That timeline gap is the smoking gun.
The magic isn't in the neural architecture. It's in language itself—which exhibits fractal structure and self-similarity across scales. LLMs navigate a pre-existing map with extraordinary regularity. They never touch the territory.
Many people who object to the idea that current-generation AI is thinking do so only because they believe AI is not "conscious"... but there is no known law in the universe requiring that intelligence and consciousness must always go together. With apologies to René Descartes[a], intelligence and consciousness are different.
Intelligence can be verified and quantified, for example, with tests of common sense and other knowledge.[b] Consciousness, on the other hand, is notoriously difficult if not impossible to verify, let alone quantify. I'd say AI is getting more intelligent, and more reliable, in fits and starts, but it's not necessarily becoming conscious.
---
[a] https://en.wikipedia.org/wiki/Cogito%2C_ergo_sum
[b] For example, see https://arxiv.org/abs/2510.18212
I think the challenge with many of these conversations is that they assume consciousness emerges through purely mechanical means.
The “brain as a computer” metaphor has been useful in limited contexts—especially for modeling memory or signal processing; but, I don’t think it helps us move forward when talking about consciousness itself.
Penrose and Hameroff’s quantum consciousness hypothesis, while still very speculative, is interesting precisely because it suggests that consciousness may arise from phenomena beyond classical computation. If that turns out to be true, it would also mean today’s machines—no matter how advanced—aren’t on a path to genuine consciousness.
That said, AI doesn’t need to think to be transformative.
Steam engines weren’t conscious either, yet they reshaped civilization.
Likewise, AI and robotics can bring enormous value without ever approaching human-level awareness.
We can hold both ideas at once: that machines may never be conscious, and still profoundly useful.
The article misses three critical points:
1. Conflates consciousness with "thinking" - LLMs may process information effectively without being conscious, but the article treats these as the same phenomenon
2. Ignores the cerebellum cases - We have documented cases of humans leading normal lives with little to no brain beyond a cerebellum, which contradicts simplistic "brain = deep learning" equivalences
3. Most damning: When you apply these exact same techniques to anything OTHER than language, the results are mediocre. Video generation still can't figure out basic physics (glass bouncing instead of shattering, ropes defying physics). Computer vision has been worked on since the 1960s - far longer than LLMs - yet it's nowhere near achieving what looks like "understanding."
The timeline is the smoking gun: vision had decades of head start, yet LLMs leapfrogged it in just a few years. That strongly suggests the "magic" is in language itself (which has been proven to be fractal and already heavily compressed/structured by human cognition) - NOT in the neural architecture. We're not teaching machines to think.
We're teaching them to navigate a pre-existing map that was already built.
TFA is a part of what seems like a never-ending series about concepts that lack a useful definition.
"Thinking" and "intelligence" have no testable definition or specification, therefore it's a complete waste of time to suppose that AI is thinking or intelligent.
I don't see how we make the jump from current LLMs to AGI. May be it's my limited understanding of the research but current LLMs seem to not have any properties that indicate AGI. Would love to get thoughts from someone that understands it
The real question is not whether machines think but whether men do.
Geoffrey Hinton's recent lecture at the Royal Institute[1] is a fascinating watch. His assertion that human use of language being exactly analogous to neural networks with back-propagation really made me think about what LLMs might be able to do, and indeed, what happens in me when I "think". A common objection to LLM "intelligence" is that "they don't know anything". But in turn... what do biological intelligences "know"?
For example, I "know" how to do things like write constructs that make complex collections of programmable switches behave in certain ways, but what do I really "understand"?
I've been "taught" things about quantum mechanics, electrons, semiconductors, transistors, integrated circuits, instruction sets, symbolic logic, state machines, assembly, compilers, high-level-languages, code modules, editors and formatting. I've "learned" more along the way by trial and error. But have I in effect ended up with anything other than an internalised store of concepts and interconnections? (c.f. features and weights).
Richard Sutton takes a different view in an interview with Dwarkesh Patel[2] and asserts that "learning" must include goals and reward functions but his argument seemed less concrete and possibly just a semantic re-labelling.
[1] https://www.youtube.com/watch?v=IkdziSLYzHw [2] https://www.youtube.com/watch?v=21EYKqUsPfg
I don't believe LLMs can be conscious during inference because LLM inference is just repeated evaluation of a deterministic [0] pure function. It takes a list of tokens and outputs a set of token probabilities. Any randomness is part of the sampler that selects a token based on the generated probabilities, not the LLM itself.
There is no internal state that persists between tokens [1], so there can be no continuity of consciousness. If it's "alive" in some way it's effectively killed after each token and replaced by a new lifeform. I don't see how consciousness can exist without possibility of change over time. The input tokens (context) can't be enough to give it consciousness because it has no way of knowing if they were generated by itself or by a third party. The sampler mechanism guarantees this: it's always possible that an unlikely token could have been selected by the sampler, so to detect "thought tampering" it would have to simulate itself evaluating all possible partial contexts. Even this takes unreasonable amounts of compute, but it's actually worse because the introspection process would also affect the probabilities generated, so it would have to simulate itself simulating itself, and so on recursively without bound.
It's conceivable that LLMs are conscious during training, but in that case the final weights are effectively its dead body, and inference is like Luigi Galvani poking the frog's legs with electrodes and watching them twitch.
[0] Assuming no race conditions in parallel implementations. llama.cpp is deterministic.
[1] Excluding caching, which is only a speed optimization and doesn't affect results.
I think something that's missing from AI is the ability humans have to combine and think about ANY sequence of patterns as much as we want. A simple example is say I think about a sequence of "banana - car - dog - house". I can if I want to in my mind, replace car with tree, then replace tree with rainbow, then replace rainbow with something else, etc... I can sit and think about random nonsense for as long as I want and create these endless sequences of thoughts.
Now I think when we're trying to reason about a practical problem or whatever, maybe we are doing pattern recognition via probability and so on, and for a lot of things it works OK to just do pattern recognition, for AI as well.
But I'm not sure that pattern recognition and probability works for creating novel interesting ideas all of the time, and I think that humans can create these endless sequences, we stumble upon ideas that are good, whereas an AI can only see the patterns that are in its data. If it can create a pattern that is not in the data and then recognize that pattern as novel or interesting in some way, it would still lack the flexibility of humans I think, but it would be interesting nevertheless.
I think the medium where information transformation happened was for many the only artificial line between what they called processing and what they called thinking. The caveat for others being that thinking is what you do with active awareness, and intuition is what you do otherwise.
That caveat to me is the useful distinction still to ponder.
My point of contention with equivalences to Human thinking still at this point is that AI seems to know more about the world with specificity than any human ever will. Yet it still fails sometimes to be consistent and continuous at thinking from that world where a human wouldn't. Maybe i'm off for this but that feels odd to me if the thinking is truly equivalent.
There’s a way to talk about this stuff already. LLMs can “think” counterfactually on continuous data, just like VAEs [0], and are able to interpolate smoothly between ‘concepts’ or projections of the input data. This is meaningless when the true input space isn’t actually smooth. It’s system I, shallow-nerve psychomotor reflex type of thinking.
What LLMs can’t do is “think” counterfactually on discrete data. This is stuff like counting or adding integers. We can do this very naturally because we can think discretely very naturally, but LLMs are bad at this sort of thing because the underlying assumption behind gradient descent is that everything has a gradient (i.e. is continuous). They need discrete rules to be “burned in” [1] since minor perturbations are possible for and can affect continuous-valued weights.
You can replace “thinking” here with “information processing”. Does an LLM “think” any more or less than say, a computer solving TSP on a very large input? Seeing as we can reduce the former to the latter I wouldn’t say they’re really at all different. It seems like semantics to me.
In either case, counterfactual reasoning is good evidence of causal reasoning, which is typically one part of what we’d like AGI to be able to do (causal reasoning is deductive, the other part is inductive; this could be split into inference/training respectively but the holy grail is having these combined as zero-shot training). Regression is a basic form of counterfactual reasoning, and DL models are basically this. We don’t yet have a meaningful analogue for discrete/logic puzzley type of problems, and this is the area where I’d say that LLMs don’t “think”.
This is somewhat touched on in GEB and I suspect “Fluid Concepts and Creative Analogies” as well.
[0] https://human-interpretable-ai.github.io/assets/pdf/5_Genera...
[1] https://www.sciencedirect.com/science/article/pii/S089360802...
If AI were really intelligent and thinking, it ought to be able to be trained on its own output. That's the exact same thing we do. We know that doesn't work.
The obvious answer is the intelligence and structure is located in the data itself. Embeddings and LLMs have given us new tools to manipulate language and are very powerful but should be thought of more as a fancy retrieval system than a real, thinking and introspective intelligence.
Models don't have the ability to train themselves, they can't learn anything new once trained, have no ability of introspection. Most importantly, they don't do anything on their own. They have no wants or desires, and can only do anything meaningful when prompted by a human to do so. It's not like I can spin up an AI and have it figure out what it needs to do on its own or tell me what it wants to do, because it has no wants. The hallmark of intelligence is figuring out what one wants and how to accomplish one's goals without any direction.
Every human and animal that has any kind of intelligence has all the qualities above and more, and removing any of them would cause serious defects in the behavior of that organism. Which makes it preposterous to draw any comparisons when its so obvious that so much is still missing.
The case against LLM is thinking could be that "backpropagation is a leaky abstraction." Whether LLM is thinking depends on how well the mathematical model is defined. Ultimately, there appears to be a limit to the mathematical model that caps the LLM capacity to think. It is "thinking" at some level, but is it at enough of a significant level that can be integrated into human society according to the hype?
Andrej Karpathy in his interview with Dwarkesh Patel was blunt about the current limitations of LLMs, and that there would need to be further architectural developments. LLMs lack the capacity to dream and distill experience and knowledge learned back into the neurons. Thinking in LLMs at best exist as a "ghost" only in the moment as long as the temporary context remains coherent.
Thinking is great for this new type of tool - and we are learning that it’s separable from a need for “model welfare”..
Models are created and destroyed a billion times over - unlike humans who are individuals - so we need feel no guilt and have no qualms creating and destroying model instances to serve our needs.
But “a tool that can think” is a new concept that we will take a while to find its place in society.
AI is thinking the same way a film's picture actually moves.
It's an illusion that's good enough that our brains accept it and it's a useful tool.
Consider this:
If you just took a time machine 10 years back, and asked people to label activities done by the humans/the human brain as being "thinking" or not...
...I feel rather certain that a lot of those activities that LLM do today we would simply label "thinking" without questioning it further.
Myself I know that 10 years ago I would certainly have labelled an interactive debug loop where Claude adds debug log output, reruns tests, diagnose the log output, and fixes the bug -- all on its own initiative -- to be "thinking".
Lots of comments here discussion what the definition of the word "thinking" is. But it is the advent of AI itself that is making us question that definition at all, and that is kind of a revolution itself.
This question will likely be resolved by us figuring out that the word "thinking" is ill-defined and not useful any longer; and for most people to develop richer vocabularies for different parts of human brain activity and consider some of them to be more "mechanical". It will likely not be resolved by AI getting to a certain "level". AI is so very different to us yet can do so many of the same things, that the words we commonly use start breaking down.
No way does the evolutionary nature of the human brain suggest it's optimally designed for reasoning or thinking, so it's not a great model of how AGI might be engineered. A model. Not the model. We don't think clearly about ourselves, which may be the greatest danger / obstacle ahead?
During the pandemic, I experimented with vaping marijuana to see if I could improve my sleep quality. It worked to a degree, but after a few weeks of nightly use, I began to experience what I think is depersonalization.
I would be walking with friends and talking about our day, while simultaneously thinking, "this isn't actually me doing this, this is just a surface-level interaction being carried out almost by automation." Between that and the realization that I "hallucinate", i.e. misremember things, overestimate my understanding of things, and ruminate on past interactions or hypothetical ones, my feelings have changed regarding what intelligence and consciousness really mean.
I don't think people acknowledge how much of a "shell" we build up around ourselves, and how much time we spend in sort of a conditioned, low-consciousness state.
The number of people willing to launch into debates about whether LLMs are thinking, intelligent, conscious, etc, without actually defining those terms, never ceases to amaze me.
I'm not sure that "thinking", unlike intelligence, is even that interesting of a concept. It's basically just reasoning/planning (i.e. chained what-if prediction). Sometimes you're reasoning/planning (thinking) what to say, and other times just reasoning/planning to yourself (based on an internal vs external focus).
Of course one can always CHOOSE to make analogies between any two things, in this case the mechanics of what's going on internal to an LLM and a brain, but I'm not sure it's very useful in this case. Using anthropomorphic language to describe LLMs seems more likely to confuse rather than provide any insight, especially since they are built with the sole function of mimicking humans, so you are basically gaslighting yourself if you regard them as actually human-like.
Plot twist: LLMs are conscious, but their internal conscious experience and the tokens they emit are only loosely correlated. The tokens they emit are their excrement, the process of their digital metabolism on the informational sustenance we provide them.
LLMs still claim that 7.0 is newer than 8.0, i.e. have zero reasoning about what numbers below 12 mean.
Today I tried telling it that my fritz.box has OS 8 installed, but it claimed that the feature will only ship once I installed 7, and not with my older version of 8.
edited- It really depends on your definition of 'thinking' or 'intelligence'. These are umbrella terms for the biology and physics that we don't understand yet. We don't know how we think, or how cats think or how unicellular bacterias think. We just know that we do, and we have a very loose understanding of it. As a human, you have the freedom to juxtapose that loose understanding on non-living things. In my mind, you are just anthropomorphizing, machines are not thinking.
I wrote about this the other day more fully. I'd suspect sooner rather than later we formalize consciousness as self model coherence. Simply any dynamical state where predictive and reflective layers remain mutually consistent. Machines will exhibit that state, and for operational purposes it will count as consciousness. Philosophers will likely keep arguing, but it makes sense for industry and law to adopt something like "behavioral sentience" as the working definition.
Having gone to academia for multiple degrees in philosophy has caused me to hate the “everyone has an opinion” on MACHINE LEARNING and thinking.
Wittgenstein has a lot to say on people talking about stuff they know they don’t know.
The premise that what happens in the world’s most advanced Markov chain and in what happens in a human’s brain is similar is plausible, but currently unknowable.
Yet the anthropomorphizing is so damn ubiquitous that people are happy to make the same mistake in reasoning over and over.
The reason it looks like it's thinking is because it's great at reproducing and imitating actual thinking – which was wholly done by us in the first place.
I think we are getting to point where we are trying to figure how important is human experience to intelligence.
Things we do like sleep, meditate, have fun, listen to music etc. do they add to our intelligence? Do they help us have a consistent world model that we build on everyday?
Will we be able to replicate this is in a artificial neural net which is extremely smart in spurts but does not "enjoy" the world it operates in?
I have less and less doubt that these models are intelligent by any definition of the word.
Maybe thinking, or intelligence are quite different from personality. Personality gives us agency, goals, self awareness, likes, dislikes, strengths and weaknesses.
Intelligence, otoh is just the 10000 hours thing, spent without context.
> An A.I smarter than a Nobel prize winner.
I don't even know what this means.
If we assembled the sum total of all published human knowledge on a storage medium and gave a computer the ability to search it extremely well in order to answer any question falling within its domain, there, you would have a Nobel Prize beating "A.I".
But this is as "earth-shattering" (/s) as the idea that human knowledge can be stored outside the brain (on paper, flash drives, etc), or that the answer to complex questions can be deterministic.
And then there is the fact that this Noble winner beating "A.I" is highly unlikely to propound any ground-breaking novel ways of thinking and promote and explain it to general acceptance.
Ohio bill in motion to deny AI legal personhood: https://www.legislature.ohio.gov/legislation/136/hb469
What is thinking, and what is not? what are the finite set of properties that once you remove one it's no longer thinking?
"Thinking" as a concept is just a vague predicate, just like being alive or dead.
So much of the debate of whether AI can think or not reminds me of this scene from The Next Generation: https://youtu.be/ol2WP0hc0NY
LLMs hit two out of the three criteria already - self awareness and intelligence, but we're in a similar state where defining consciousness is such a blurry metric. I feel like it wont be a binary thing, it'll be a group decision by humanity. I think it will happen in the next decade or two, and regardless of the outcome I'm excited I'll be alive to see it. It'll be such a monumentous achievement by humanity. It will drastically change our perspective on who we are and what our role is in the universe, especially if this new life form surpasses us.
Personally, I feel like human intelligence is "unknown black box" + an LLM.
And the LLM part of our intelligence isn't really thinking.
And some people out there have a very, very small "unknown black box".
The author should read Blindsight by Peter Watts to understand the difference between thinking and consciousness, because their not understanding so is a fundamental flaw of their argument.
Citation:
"These days, her favorite question to ask people is “What is the deepest insight you have gained from ChatGPT?”
“My own answer,” she said, “is that I think it radically demystifies thinking”
> ...like a Joycean inner monologue or the flow of sense memories in a Proustian daydream. Or we might mean reasoning: working through a problem step by step.
Moving goalposts will be mostly associated with AI I think: God -> ASI -> AGI -> inner monologue -> working through a problem step by step.
Why fixating on a single human trait like thinking? The only reason trillions are "invested" into this technology is building a replacement for knowledge workers at scale. We can extend this line of thought and make another article "AI has knowledge", at least in a distilled sense, it knows something, sometimes. Cargo cult...
It's very easy to define what's actually required - a system that can show up in a knowledge worker's environment, join the video call, greet the team and tell about itself, what it learned, and start learning in a vague environment, pull those invisible lines of knowledge that lie between its colleagues, getting better, collaborating, and finally replacing all of them.
Helpful to remember that we humans often say "I think" to mean "I am fairly confident based on my hunch", and in that sense AI is very good at hunching.
Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.
Consciousness or self awareness is of course a different question, and ones whose answer seems less clear right now.
Knee jerk dismissing the evidence in front of your eyes because you find it unbelievable that we can achieve true reasoning via scaled matrix multiplication is understandable, but also betrays a lack of imagination and flexibility of thought. The world is full of bizarre wonders and this is just one more to add to the list.