alt Hacker News> There's also a need to consider the rights that this new intelligence should have
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
Replies
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?