The analogy I make is between airplanes and birds.
Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly.
Similarly, current AIs are not alive, are not conscious - but think.
All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.
There are birds that go far longer than typical aeroplane flight times without a single flap of their wings either using thermal, ridge, or other sources of lift. Are these flying birds? I've shared thermals with eagles flying the same circles, neither one of us flapping our wings but making minor adjustments for the same goal.
An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.
Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?
Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.
What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.
"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra
Okay sure. But given we don’t know what consciousness comes from, we shouldn’t be too glib about there being a grey area here. Historically people have made racist and speciesist judgments towards other being by assuming certain inferiorities despite obvious “thinking” happenings.
I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?