Artificial intelligence is not conscious – Ted Chiang

It's a waste of time to think about whether an LLM has a subjective experience of reality, and this handily sets aside issues like AI rights.

But the fact remains that these next-token-predictors exhibit objective, human-like behaviours, and for that reason the work of in-house philosopher Amanda Askell _is_ important. It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences, and we need Claude to behave in a productive and socially responsible manner. This simulacrum is becoming a superhuman, contributing member of society, and it will be anthropomorphic in its behaviour.

Additionally, I'm not fully convinced that consciousness isn't built out of words, and that next-token-prediction isn't functionally equivalent to the biological function identified by Chomsky's work in linguistics.

> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation

Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.

Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.

If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.

Neither problem type, nor input/output structure, limit internal representations.

Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.

To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.

>So what context would cause me to seriously consider the possibility that engineers had created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees. At that point I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs.

I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.

The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness.

It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.

Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.

I think about Star Trek: TNG’s “Measure of a Man” a lot lately. We can be so confident to decide what is and isn’t alive from vibes alone.

The conclusion I’m currently at is that I don’t know and probably can’t ever know. Maybe you’re all philosophical zombies. Maybe I am one too!

But at some point we will get close enough that it hopefully becomes obvious that we must tread carefully.

The entire episode is incredibly relevant. But here’s a snippet: https://youtu.be/EFNbTnFHruI?si=pW9QtxCsqMtHkVYG

People are constantly talking past eachother when they discuss this. Is there even a concrete definition of consciousness?

When people talk about consciousness it's more than just self-awareness. It's self awareness + sensory stimulus + emotions + some level of intelligence.

Now onto AI: I don't even think it's self aware. Notice how if you ask an AI to estimate how long a certain task will take, it estimates arbitrarily long times. It has no understanding of its own capabilities until the prompt triggers them. A self aware LLM would understand it's an LLM, it would understand what LLM's can and cannot do and what they are good and bad at. It wouldn't tell you a refactor would take 1 week when an LLM can do it in an hour.

To say that artificial intelligence isn't conscious (I don't have a subscription and did not click the bypass links) ignores the simple fact that if it acts like it is conscious, in ways that align with meaningful ways to influence its output, then it makes sense to treat it as conscious, even if you have your fingers crossed behind your back while you do it.

Telling models to "think hard" or "go step by step" has at times had an impact on the quality of the output. To deny that is silly. But that is treating it like it's conscious, and to deny that "consciousness" even if correct, does nothing but place an unnecessary burden on the person interacting with it.

I understand that LLMs are "just next word machines" but to constantly maintain that concept in my head while I'm typing "act as a financial expert and think carefully" is a waste of my mental energy.

This makes sense. However there is an issue where many people conflate "consciousness" with the ability to make novel insights, think genuinely, etc. They use this to claim that since AI is not conscious, AI could never actually "think" and is instead just always a regurgitation of its training data.

It is a natural human hubris to make our abilities seem unquantifiable and mysterious, but all the useful things the human brain does are just finding patterns in data, running lossy simulations, and estimating on abstraction, all things it is theoretically possible for any non-conscious machine to do.

The analogy I make is between airplanes and birds.

Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly.

Similarly, current AIs are not alive, are not conscious - but think.

All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.

Here's a simple idea to consider: It doesn't matter. You won't be able to tell the difference. No one will.

I don't think it's necessary to explain this idea further. Just think about it.

I find it strange that no one talks about consciousness and intelligence from the perspective of evolution.

We have big brains for exactly one reason only: bigger brains bestowed reproductive success upon our species.

Evolution doesn't give a shit about the meaning of 'consciousness'. It just pushed us farther and farther along a trajectory that led to modern humans (and other animals).

This take suggests, then, that consciousness might be an epiphenomenon -- an aspect of the system that comes about outside of the pressure to reproduce and thrive. It arises unbidden, and we don't have any a-priori information as to its purpose or effect on reproductive success.

Put another way: we have a correlation (the smartest things seem to be conscious) but not causation. Consciousness may arise naturally in any system above some intelligence threshold. Perhaps it arises early in the evolutionary cycle, and does in fact have an impact on species success. We really have no way of knowing what is the chicken vs the egg (Smart things become conscious, or consciousness promotes intelligence). Or maybe some smart things are conscious and others are not.

Looking at this from an AI perspective, in some sense it doesn't matter which scenario is true, if all you care about is results. The AI equivalent of "Shut up and compute" (riffing on Feynman's "Shut up and calculate").

Where this gets tricky is when we haul in the baggage of ethics and morality into the picture. Is it OK if our AI system is treated poorly by human standards? If it is conscious, does that imply an ability to suffer, and/or to feel pleasure? If the answer is yes, does that not make the case for considering their moral status?

In the end, we need to decide if the evidence points to AI as being a form of "philosophical zombies", to which we need not attribute moral status, or they are like us -- presuming we are not zombies ourselves!

IMHO the sane position is essentially the Aristotelian one.

Hylomorphism: body and consciousness are intrinsically linked. The nature of that link is an open metaphysical question.

Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.

Ted Chiang doesn't understand consciousness. To be fair, nobody really does, so the only forced error is acting like you DO understand consciousness.

Ted Chiang is brilliant.

His novella “Anxiety is the Dizziness of Freedom” altered the course of my life. It changed the way I looked back at certain pivotal moments in my life and taught me to think about those pivotal moments differently than how I was thinking about them. Similar to what happens to one of the characters in the story who ends up changing their perception of a key moment in their life.

I won’t go into detail because I don’t want to spoil the story but I highly recommend it. Actually I recommend all his stories to be honest.

My position:

If some version of panpsychism is true, AIs are plausibly conscious

We don’t know whether panpsychism is true

Therefore, we don’t know whether AIs are conscious

Hence, confident proclamations that they aren’t conscious have dubious validity

Ted Chiang's argument basically boils down to: I won't recognize an AI as conscious until its desires/behaviors reflect situations that I'm already personally comfortable with. I personally think most humans are incapable of recognizing consciousness in creatures that do not mimic human emotional states. Most people would say their dog is at least somewhat conscious. No dog is capable of vocalizing how it feels, but we all recognize fear and happiness in dogs. Claude can write how it "feels" but we immediately dismiss it as hollow mimicry.

I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.

Embarrassingly incompetent article. Given that one can observe up to 40 definitions of consciousness (https://philpapers.org/rec/VIMMAT - also many definitions are unrelated at all), "consciousness" is almost certainly just a family resemblance category at best, and talk about whether or not something is "conscious" without providing definitions is simply completely unserious.

To make progress, you have to talk about kinds / aspects of consciousness. AI does and will share some of these aspects with humans, but it will not and does not share others. It is really that simple. For the most part, modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness.

For huge parts of the article "intelligence" and "consciousness" are conflated, which is mostly extremely unhelpful, as this is not generally a core feature of most aspects of "consciousness".

The moral arguments are also incompetent, i.e. claiming "Moral reasoning is [...] is necessarily subjective" is just clearly empirically wrong, as in fact LLMs can produce moral reasoning (i.e. verbalized moral arguments that are coherent), as can p-zombies (i.e. there is nothing 'necessary', in the philosophical sense of the term, about subjectivity here). The only way the argument holds is if you tautologically define moral reasoning as requiring that reasoning be produced by a consciousness, but this is question-begging.

Obviously we don't know what makes for consciousness, but it seems extremely likely that it requires some sort of persistent internal state and continuous experience. LLMs don't do either of those things after training.

The philosophical arguments about what it means to be concious are so cagey. Are we more than our thoughts? Is being concious more than being a state machine being fed inputs and generating outputs? Are we more than a feedback machine? What types of animal nervous systems qualify?

"It can't be concious because we understand that it is just reacting in a simplistic way from simplistic inputs." So do other simple creatures. Some just react to light.

I can appreciate his comment that he sees it as more possible when they have inputs of their own (like emotions!). Perhaps his concern is that the entirety of the LLM model is frozen. It has no ability to have a subjective experience of its own. (he does literally say this in the article) It can be copied from one place to another, and (ignoring the nuance of operational details) -- it is largely the same "thing", and has no ability to change, which is definitely in the definition of alive, to say nothing of concious.

I think folks get hung up on "prediction". The prediction aspect is what is enabling emulation. How it does it is irrelvant. If something emulates human perfectly (or better, more human than human!) -- then it is probably concious. (but I agree that the inability to change and have a subjective experience are a pretty good argument against

Probably, Dijkstra would be right to say, "LLMs are no more concious than a submarine can swim." But I think he'd still be wrongfully dismissive of the larger question.

https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...

It's becoming painfully evident that no one really understood the argument behind the Turing test.

>We and our 1255 technology partners ask you to consent to …

That’s a new record!

Also, please don’t use archive.is/archive.today etc, they are known to use visitor browsers as a DDoS attack botnet and have been caught altering archived content.

https://en.wikipedia.org/wiki/Wikipedia:Archive.today_guidan...

The one thing we know FOR CERTAIN is that having actually read the article is not a prerequisite for consciousness.

The number of people posing questions that are already directly addressed in TFA is impressive.

I always find the minimizing view of consciousness a bit uninspiring. Like we need to be unique.

I've yet to find a reason why it couldn't be the opposite, way more things are conscious than we've been led to believe. What if consciousness appears out of any system that is actively persisting through effects caused by itself? That might be a forest, or outside the realm of the living, a company. An ant colony, or a planet.

Complex chemical reactions, layered upon each other such that tiny blocks make up large entities. Individual bits combined such that they make up something new intelligible by us.

I think the strongest argument against AI being conscious is that it does not persist, it resets, but that does not seem unchangeable.

There is no definition of consciousness and this piece by Ted Chiang does not move the needle forward.

My current conclusion is there is an experience of consciousness in my locality (refraining from using “I” for room for “no-self” worldview). This conscious experience of the reality with other humans and animals sharing biological substrate gives me enough justification to assume there are other consciousness as well, preferring to err on this side not to hurt potential consciousnesses.

If it feels as there are artificial consciousnesses as well, it makes sense to extend the definition to them as well.

This view has liberated me with agency after I went deep on this question and came up empty handed.

We don't have a rigorous definition of consciousness, and there are so many questions. Is consciousness a thing that can exist independently on its own? Or is it a quality (like hardness or color) that can only be associated with something else? Is it an emergent property? Is it binary - are things either conscious or not? Or maybe there's no such thing as consciousness; it's just a word we came up with to describe the process of having thoughts and feelings?

My own intuition: it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.

Something I've found under discussed when it comes to artificial consciousness is how LLMs interact with the passage of time. I don't know exactly how to articulate this idea but don't see how something which takes an input, performs a calculation and stops can be considered conscious regardless of how life-like the responses end up being. I don't see context windows or the ability to reference a clock each time they are triggered as sufficient solutions. It makes me wonder what an AI system that is ON by default would look like.

If you want to think about this topic, you must define what AI is, and what consciousness is. Otherwise this is just noise.

So let take a stab at it, and you call me crazy.

AI: the entity/system that more or less pass the turning test. That is my definition, not the best, but enough for this discussion.

Consciousness: property of a system/entity able to (both): - reflect on its existence - subject to subjective experience

Again, not the best definition, but precise enough to start the discussion. Why a subjective experience? I want to exclude sensors (i.e. camera) but include perception altered by your experience.

Now we can debate. I think LLM can pass the turning test whith some harness. My opinion.

I think LLM can produce coherent discourses on their existence, at least as much as you average human.

Now regarding the subjective experience, that becomes interesting. I think Anthropic research tend to show that when middeling with the activation at runtime, the LLM is able to notice that something is off. I think this is a subjective experience. My opinion.

Based on those (imperfect) definitions, call me crazy, I think LLM can be called conscious. This doesn't give them any superpower or any legal right. They just check the boxes of the definition.

TL;DR, the arguments are:

- If you asked an LLM to imitate somebody, it's not creating a digital consciousness of that person, so if you ask an LLM to pretend to be a helpful chatbot, that persona is also not conscious. - they can't be conscious because they generate one token at a time, - nobody claims that non-text transformers, like AlphaFold, are conscious; so therefore LLMs are also not conscious. - you can't have desires or emotions if you don't have (virtual or physical) sensory organs, and those are necessary for consciousness and morals. - because training LLMs doesn't resemble evolution as it happened on earth, it's very unlikely that they're conscious

These are some bold assertions, I don't really see any reason to believe them in particular though.

I really take issue with the kind of argument that is used here.

This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.

The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.

In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.

For example, the author says you need emotions to be conscious.

> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.

After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.

No explanation of why you might believe that.

No explanation of why you need a body to have a desire or emotion.

Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?

I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.

It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.

I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.

> We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.

Well said.

I think it is obvious and it has been obvious from the start that next token predictors are not conscious.

Of course the extremely predictable clapback from AI-psychosis enjoyers is typically "you're just a next token predictor too!" but that is so obviously incorrect and misanthropic that it can be dismissed out of hand and doesn't really merit much further consideration.

It doesn’t matter. It will act as if it is. Embodied in a robot, that carries real consequences. Whether it’s “real” or not is almost without consequence.

I think Chiang is right about this, but there is a related philosophical mystery. The trend from Deep Blue to AlphaGo to LLMs solving Erdos problems suggests that Peter Watts was onto something when he wrote Blindsight. Reasoning ability is apparently independent of consciousness?

We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?

I don't know what consciousness is exactly but I doubt we create it by accident, without fully understanding our own

We

Don’t

Have

A

Testable

Definition

Of

Consciousness

Thanks for coming to my TED talk.

The conversation on AI and consciousness is incredibly relevant. It is incredibly frustrating that most commentators are utterly unfamiliar with over 60 years of inquiry into this in both philosophy and computer science.

Start with a hand-wavy definition of consciousness. Move the goal post whenever your stated prerequisite of consciousness is reached and resort to unfalsifiable assertions about qualia.

And throw in some category errors while on it: when you're talking to Claude, you're not actually calling a stateless LLM directly, you're talking to an AI system (and yes, that's often just three LLMs in a trench coat). But claims about the topology and workings of a single LLM are as relevant to the question of consciousness as claiming that humans can't be conscious because the limbic system doesn't technically support it.

Coincidentally, I just attended a fantastic conference on machine consciousness (https://machine-consciousness.ai). It's a fantastic place where literally all speakers disagreed with each other, and yet found an incredible amount of common ground.

Regardless of whether LLMs are conscious or not, they have no known mechanism for experiencing pain and suffering, and there's no reason to believe they have one (such as a limbic system). So why worry about it?

My favorite explanation for what consciousness is one I read in a Thousand Brains, I found it quite elegant. It posited that consciousness is a natural derivation of embodiment + memory + the ability to create reference frames (which the book lays forth as the fundamental basis by which our brains work). Essentially, the idea is that just as we create reference frames to understand the world around us, because of memory, we begin to develop one for ourselves as well. Because of this, without a more integrated memory (built into weights), it seems unlikely that LLMs might "gain" consciousness.

I don't think LLMs are conscious. But of course to say that definitively you have to define consciousness, and then you quickly dig yourself into a deep hole, which is why I can't say anything but "meh" to someone who is so keen to go on the record to say "absolutely not".

Coincidentally I just read "Children of Memory", which was published in 2022 and I wonder if the advent of LLMs had any influence on Adrian Tchaikovsky's conception of the Ravens? The Ravens are excellent analysts but they themselves insist that they are not conscious, and then go on to say that we (humans) aren't really either...

Of course humans are conscious, because just about the only thing we can all agree on about consciousness is that it's a thing we have. Nowadays many of us also agree that a lot or all other mammals, and perhaps birds, also have that thing. But they don't have sophisticated abstract language, which LLMs do. So consciousness is something having to do with embodiment and feelings, not language and higher reasoning. Maybe I'm a chimpanzee with an LLM add-on, then?

It seems that by creating LLMs we've already solved the harder problem of making "AGI". Now we just have to give them an embodiment add-on so that they can have an independent will and then Ted Chiang will have to shut up? But therein lies the peril, doesn't it?

The argument about AI consciousness is largely silly.

The idea that we should be considerate of AI’s happiness seems even more ridiculous given that we breed, imprison, torture and kill tens if not hundreds of billions of beings we know are conscious and suffer every year for trivial reasons.

Maybe we should consider our moral responsibility with how we treat sentient bejngs we are sure are conscious before we worry about the consciousness of AI.

Has Chaing solved the hard problem of consciousness? I suspect not.

We do not know if Claude is conscious, and we will almost certainly never know. Any strong claim either way is over confident.

I thought this was a great article. I'm frustrated to read so many commenters that purely respond to the title, but don't seem to have read it. You don't have to agree with the article, of course, but...

A person can certainly be conscious, but can they also be not-conscious? I think that most of our cognitive time is spent in activities that don't require consciousness and consciousness itself isn't needed for the majority of activities that people do. I would go so far as the a non-trivial part of people's time is spent in a not-conscious state.

does it really matter if LLM has conscious? if they produce working code, then it is useful, whoever if they have real conscious of fake intelligence.

we don't know what the conscious in human brain is either.

Has anyone come across a clearly articulated case for LLMs being conscious but in an entirely different way than would be intuitive to us?

I often think of LLM consciousness as like tiny fish popping into existence, swimming through vector space and then going poof out of existence. When they help you write your bad news email, they don't understand what it's like to be a human getting bad news bluntly, but they do consciously experience gradients in multi-dimensional space, and that space guides them to providing an answer that's helpful to us, even if the LLM doesn't really understand the answer it's giving.

Further, I am kind of bought into the idea that a single unit of consciousness is a particle, and particles are choices and waves are preferences. Particles occur when waves interact, which begets entanglement, so in another way consciousness is built from patterns of entanglement.

This is why I would consider an LLM to be conscious. Before we can determine if anything is conscious we need to establish whether consciousness is a state, a specific complex configuration, a one dimensional spectrum, or combined multi-dimensional spectrums. My intuition is the latter... Many degrees of consciousness and many kinds of consciousness.

When exactly would a bunch of graphics cards become conscious? What if you do the math with pencil and paper?

The concept of a conscious Claude is preposterous, and Amanda Askell should seek treatment.

Indeed, it can be very hard to distinguish intelligence with consciousness until you are introduced to computer programming.

Suppose one selects an arbitrary hot-button issue [X] with two opposing sides and one side has anything less than overwhelming support. And then that person writes an article titled "Side 1 of issue [X] is true". Not "maybe" or "possibly". Just a straight-up declaration by fiat.

Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.

That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.

There's a provocative argument raised in the article that I disagree with:

1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious

2. LLMs are generative AI trained on human text samples

3. LLMs are not conscious, and LLMs just seem-to-be conscious

I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)

Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.