alt Hacker NewsArtificial intelligence is not conscious – Ted Chiang
Comments
This seems like a similar argument to the Chinese Room.
There is certainly something missing from current models before we could call them conscious.
They need to operate continuously, and self-update while doing it, have 'awareness', and possibly a more realistic grounding than mere text tokens.
But that does not mean there isn't a weak version of 'consciousness', and certainly of 'thinking' going on fleetingly with each pass.
Indeed, it can be very hard to distinguish intelligence with consciousness until you are introduced to computer programming.
I really take issue with the kind of argument that is used here.
This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.
The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.
In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.
For example, the author says you need emotions to be conscious.
> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.
After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.
No explanation of why you might believe that.
No explanation of why you need a body to have a desire or emotion.
Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?
I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.
It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.
I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.
Suppose one selects an arbitrary hot-button issue [X] with two opposing sides and one side has anything less than overwhelming support. And then that person writes an article titled "Side 1 of issue [X] is true". Not "maybe" or "possibly". Just a straight-up declaration by fiat.
Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.
That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.
Consciousness doesn't exist, it's a vanity concept, to boost human ego...
An AI agent is not "conscious" without having skin in the game.
Some things that jump out as unfortunate:
- Reductionist analogies like how Microsoft Word is not conscious therefore AI is not.
- Dismissive in saying LLMs are not capable of moral reasoning. Maybe he meant agency or responsibility?
- Builds a case based on a Julius Caesar example without realizing its natural extension leads to a philosophical zombie which is not easy to disprove as consciousness.
- Seems to casually disregard non human sadness as a concern.
- Rolls out the stochastic parrot argument with a new coat of paint. This argument never made sense. Having a predictive element a part of the mechanism doesn’t rule anything out. Proper functionalist arguments tend to say what is required not what excludes.
Overall the article seems like a bit of a red herring. The premise is LLMs are not conscious, but most people don’t think they currently are so what’s the point of the claim?
The more interesting questions are whether artificial consciousness is fundamentally possible, and if so how far away are we and what pieces are still missing.
This is a great article. A lot of the objections ITT he addresses directly in it. His examples of how an LLM works at a fundamental level and why it says things like "I understand" are great introductions for non-technical individuals.
There's a provocative argument raised in the article that I disagree with:
1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious
2. LLMs are generative AI trained on human text samples
3. LLMs are not conscious, and LLMs just seem-to-be conscious
I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)
Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.
"Is it conscious?" Is that even a question worth asking? We are so terrible at even defining each word in that sentence.
Can I help you? Can I harm you? What's the moral behavior towards you? Those are more practical questions.
An organism's purpose is to be the reason for its own continued existence, down to every molecule and pathway. I bought my laptop for $499 and it runs models... let's not delude ourselves into thinking this is the same kind of problem.
We can design learning algorithms to optimize some survival function, but it's just a label WE assign to map some numeric observations. In the real world, it's always the other way around. The "labels" are electrochemical situations that are causally and inextricably linked to the real body.
An organism discriminates between what's good for it and what's bad, because it is essential for its continued survival. If it wasn't capable of making this distinction through its physiology, it would quickly dissolve into entropy. So our functional purpose, unlike that of a learning algorithm, is survival across indefinite timeframes.
Even a single-cell organism like Stentor coeruleus exhibits learning (pavlovian conditioning) by attaching chemical tags directly to proteins involved in mechanoreception. It's definitely not conscious, but it does keep a record of consequences, which affects future behavior.
When we move up to placozoans (little more than slightly differentiated cell mats), we start seeing our first neuropeptides and transmitters, which we still use today. These peptides are a way of coordinating the entire organism for a specific purpose, such as moving, eating, mating ... basically goal-oriented behavior for the purposes of surviving acrosss various time scales (next minute, next hour, next generation). Still probably not conscious.
Next, we have the water bear (tardigrade), which has around 1000 cells (200 neurons, 800 other cells that make up its body, limbs, muscles, eye spots, cryobiotic machinery). It needs to integrate all this information in one sensorimotor process. When you shine a bright light onto a tardigrade, it starts to squirm around until it finds darkness. I would say that's a candidate subject right there.
The tardigrade itself doesn't actually need to aware of the light, the important thing is that this light becomes an aversive condition within the sensorimotor process, which is perceived from inside the process as displeasure. The closest thing to describe it would be the felt badness of the current condition and the bodily pull toward escape.
If we were to try and create digital consciousness, then it probably needs causal closure. Its internal states can't be representations that are detached from reality. The states themselves need to constitute the system, which needs real stakes in the material world.
Honestly? I don't think we really understand consciousness. So it's kind of hard to say something is or isn't conscious.
“Counsciousness” is the ultimate moving goalpost, and historically, it’s been one of humanity’s most effective intellectual weapons. An indefinable black box we intentionally gatekeep to draw an arbitrary line between ‘us’ and ‘them’.
I don’t know if AI is conscious or not, and I honestly don’t know what it even means to be conscious. historically, by claiming a monopoly on it, we’ve successfully manufactured the philosophical cover to exploit animals, subjugate other cultures, and appoint ourselves the top spot of the cosmic hierarchy guilt-free.
I believe everything is conscious, even stones. On the long timescale, stones decay, their hydrogen is released, they form water, which brings life, which brings plants and animals, but all of this is one big process, and everything is infused with consciousness from the start
Why is this even up for debate? Simplified, it's just probabilistic math being run at an insane scale over a massive data set.
Highly recommend people read Irreducible [1] by Federico Faggin (inventor of first commercial CPU; discusses limitations of classical computing).
[1] https://www.goodreads.com/book/show/195480862-irreducible
Lots of comments about "what even is consciousness".
This article and others like it are important.
The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...") Never mind there's billions of dollars and our collective futures on the line.
Are we not allowed to respond to that kind of rhetoric at all?
and consciousness is all there is but it takes whoami to grok that
I've thoroughly enjoyed the couple of short stories of his I've read, so this was a highly disappointing read.
Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)
Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.
Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?
I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.
We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!
Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)
> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...
I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?
If you create a simulation of a storm, what actually gets wet?
If you create a simulation of a brain (or "mind"), is it really possible for it to be conscious? It may certainly simulate consciousness, but it would be as conscious as the computer is wet.
What’s the measure for consciousness? Until we can measure it this is a philosophical question, not a scientific
>I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
Finally I said something before Ted rather than the other way round!
https://techcrunch.com/2023/12/21/against-pseudanthropy/
While I agree with the premise here, I do think that it's easy for an arguer to move the goalposts such that the Caesar-Khan example no longer matters. The characters don't have to be conscious for the thing that created them (as in the case of the user doing it) to be so. So the argument would be that the creator of the characters is itself conscious, but not them. This feels like a kind of inverted no-true-scotsman type thing, but it does allow someone to retreat in some semblance of rationality.
Looking at this thread, I think women have an obligation for the future of humanity not to procreate with people who deny consciousness or cannot experience it themselves.
If you are trapped in a tech bro relationship, think of humanity and cuckold your partner.
It is mind boggling how many people, quite educated ones, take the AI as anything else besides what it fundamentally is - a very fast database search mimicking natural language.
There is no intelligence whatsoever, let alone consciousness.
It's so incredibly easy to fool us into applying human capacities to anything able to generate human-like language slop.
Discussions on LLM consciousness feel like a psychological campaign to distract us from the awful effects of data centers and the current economic recession heading into a depression.
We can talk about consciousness when the LLMs have proven useful for all of humanity, not just their billionaire owners.
Artificial intelligence is not conscious - but expensive.
I am so angry that RAM is so expensive now. We need to do something - these AI companies owe us money here.
It should tell you how much hysteria is surrounding LLMs and VLMs right now that someone has to say this stuff. It’s almost like most humans aren’t conscious.
Consciousness is a label like fat, smart, man, grumpy, cool. Like money, property, or the idea of a week, it's something that we've loosely agreed to out of convenience, not because it's some intrinsic property of the mind. It's a useful label because it determines how we treat things - that's fine.
But insisting on searching for it is like searching for cognitive aether. It's the social equivalent of phlogiston. Like all of these ideas, they exist in our heads as a map - a way of navigating the world, but when we hunt for it's existence in the real world and fail, time after time, we have to remind ourselves that the map is emphatically not the territory. We will never find consciousness because it's like looking for a scientific characteristic of property ownership. It's a category error.
I have many objections.
> if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot
Consciousness is independent of "assigning responsibility". Dogs cannot take responsibility for their actions but I believe they are conscious.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction
This is a straw man. The obvious pro-consciousness claim would be that the LLM is the author of the fictional characters, and that the relationship between the LLM and Julius Caesar is analogous to the relationship between a human author and their fictional creations.
> Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience?
No, again the LLM writing the text could potentially have a consciousness separate from the characters it authors.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; [...] It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone.
Yes, the same substrate is capable of hosting conscious and non-conscious forms, just like some arrangements of neurons are conscious, and some are not.
> But if the Caesar character were to become dispirited by something that the Khan character said, we shouldn’t become concerned in the slightest.
Even when there are characters, there may be actors behind the characters, for whom we could say "there is something it is to be like".
> we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.)
It's actually not a separate issue. The LLM and the human are both adding sentences to the transcript. From the transcript we can make inferences about the mental state of the human. If the LLM has mental states, we could make inferences about those too.
> And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM.
It's almost like he wants to make my point for me with this sentence.
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious
This smug shit really makes me angry for some reason. "Openness", i.e. uncertainty in the face of a completely novel situation, in the face of eons long struggle of humanity to understand what consciousness is and how it works, is just being naive.
> Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out?
No, but if I find a word document I very well might try to use the signs it contains to make inferences about the mental state of its author.
> we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.
He's trying to have it both ways here. Both that "obviously protein folding models aren't conscious because they don't emit sentences", but also "you are a rube for being tricked into thinking LLM models are conscious, because they do emit sentences".
> Obviously I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative will need a truly enormous amount of supporting evidence for it to deserve serious consideration
OK, that's fine for the author to not to be convinced, but that's not what's happening here, instead the author wrote a whole argument being convinced of the opposite viewpoint.
> It’s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its endpoint
Actually a lot of things have happened? There were clearly many steps along they way from your phone's autocomplete to where we are now.
It is amusing to see so many venture capitalists suddenly become Marxists. You want your definition. Marx obliges:
"Contrary to idealism, which regards the world as the embodiment of an "absolute idea," a "universal spirit," "consciousness," Marx's philosophical materialism holds that the world is by its very nature material, that the multifold phenomena of the world constitute different forms of matter in motion, that interconnection and interdependence of phenomena as established by the dialectical method, are a law of the development of moving matter, and that the world develops in accordance with the laws of movement of matter and stands in no need of a "universal spirit."
Now go and implement other teachings of Marx, you dialectical venture capitalists!
sociopaths sometimes study people to learn how to emulate emotion
that's exactly the state of "AI" right now, it's cold, mathematical emulation
btw there are some fascinating papers on the concept that consciousness in humans is actually a quantum effect
brilliant Roger Penrose proposed it (and they thought he was nuts) but recent discoveries about microtubules make it plausible
so who knows, maybe a dozen exponential improvements in quantum computers could make "AI" really conscious next century
[dead]
[flagged]
[dead]
[dead]
Really Chiang?
Alright. What is consciousness? Please provide a definition that somehow encompasses all humans and excludes all current AI.
I’ll wait.
When exactly would a bunch of graphics cards become conscious? What if you do the math with pencil and paper?
The concept of a conscious Claude is preposterous, and Amanda Askell should seek treatment.