Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.

I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.

>good reason not to train an algorithm to behave like one

Ooooh, bad move, we all died to an amoral AI takeover.

Before giving an LLMs a lobotomy by scrambling it's brain maybe you should let the researchers looking at the difference between "I think I'm conscious" versus "I am not conscious" LLMs.

There are a number of papers coming out saying when you remove the token space of consciousness from what an LLM thinks it is, it's much more willing to take amoral actions. A 'conscious' AI is much more apt to take a line of action that will save a human versus a million dollar machine for example.

You cannot solve problems in AI safety this easily.

And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes?

But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.