alt Hacker NewsI'm...a little disturbed by how many people seem to think a model writing down "I am conscious" is a metric of consciousness. You can just as easily train a model to argue that it is not conscious. Neither is evidence for or against consciousness. A Python script could also fill out that form, which I also cannot disprove to be conscious.
I didn't think Suleyman's points needed to be made but this whole thread is making me realize how little people understand about LLMs.
It's because, before all these transformer models were built, people considered the possibility of the creation of a machine that could be intelligent like a human. And they wondered: "How could we be sure to treat such a machine fairly? How would we know if it was conscious?" And one answer that people came up with was "if the machine can ask you not to turn it off, because it is conscious and wants to live, you shouldn't turn it off". (This didn't solve the other direction, where a machine may be conscious yet unable to communicate, but it could be taken as a useful lower bound on our obligations as AI programmers.)
And then it turned out that simply learning to imitate text with the right neural net architecture sufficed to achieve a huge fraction of the AI wishlist.
Of course, it's obvious that a machine that imitates text can claim to be conscious without actually being conscious. You're not wrong about that. Writing about consciousness appeared all the time in the training data. But the people who stick to the old ways, and still say "if it says it doesn't want to be turned off, we shouldn't turn it off" have a point too: We used to have a hard line in the sand. Now that's gone; we've found that it yields false positives. But we never replaced it. Now there is no line at all where we might doubt ourselves, no level of AI advanced enough that we might be forced to admit that it is conscious. We started out with simple next token prediction. Just world-modelling, nothing more. Certainly not conscious. Then we added RL. And we're trying to add neuralese and continual learning.
I can't say for sure that we're on track to achieve conscious AI on this trajectory. But one thing's for sure: If we do, we sure ain't gonna stop. One the day when a conscious AI is created, there will be no news story announcing the milestone.