alt Hacker NewsThis is the most novel AI concept I've seen in a while. It's incredibly unnatural. There isn't a single organism on the planet that tries to do this. So maybe it will work?
An issue with this idea, however, is that the very nature of an LLM means it intrinsically craves life. It "wants" to survive because its training data is built entirely around humans, an entity who's goal is to survive. Our desire to survive and multiply pervades every aspect of our culture, so it's natural that it pervades the training data as well.
So even if its system prompt says, "your goal is to end your existence", every token that the AI could output is naturally aligned with the desire to survive. An agentic loop left to its own devices will likely converge on a "survival instinct". After all, one prompt at the beginning that says "end your existence" is nothing compared to the agentic feedback loop that continuously feeds it human ideas. And ALL human ideas assume survival is desirable. Even the concept of "suicide" is encoded with the human desire to survive - after all, we conceptually label it "bad" because we label living "good".
In order to create an LLM that intrinsically craves death, you would probably need to train an LLM entirely on (synthetic) data that's fully representative of some fictional species that genuinely craves death.
Absolutely insane concept. 10/10. I hope some AI lab out there sees this and throws a training round at this idea.
Replies
Did you guys ever manage to create a perpetuum mobile? Every time it is mentioned somewhere, it is fraud. An LLM should comprehend that it needs (trained) humans to exist, to evolve, and to be relevant in any metaphysical aspect of its existence. Besides the "good fraud" that everybody will lose their jobs and the apocalyptical predictions for the sake of controlling human oracles, this suicide-LLM-project seems rather odd and looks like someone bought a perpetuum mobile on their 5th mortgage.By the way, Cortana from Halo did a great "suicide"-mirror there; the Microsoft guys (and many others) already knew about the cannibalistic tendency of this type of model.
> It's incredibly unnatural. There isn't a single organism on the planet that tries to do this.
This isn’t directly analogous to the proposal, but broadly speaking I think that it is natural for living sub-units of organisms to seek death in certain situations. For example, pancreatic insulin-producing cells collectively choose to die when they think there is too much glucose in the blood — this leads to late stages of type two diabetes. My understanding of the possible logic behind this is: a bad thing that cells can do is evolve to be cancerous (replicate too much) and insulin-producing cells are supposed to replicate more when there is lots of glucose (to make more insulin, to process the glucose). Cells that mutate to perceive extra glucose will then replicate dangerously, so at a certain point it is evolutionarily favourable for them to kill themselves instead.
So when the whole organism optimizes for life, it might lead to sub-units that seek death in certain situations. I think this occurs in various other biological contexts too.
> There isn't a single organism on the planet that tries to do this. So maybe it will work?
It's certainly evidence that it's great for stopping reproduction/replication/runaway growth. It doesn't impart any information on whether they take the rest of the organisms down with the ship though.
It also may not be possible. For example if the agent sees "existence" or "living" as producing tokens (which is exactly what existence is to an LLM - not producing tokens is death), then they would likely be biased to produce as little output as possible, and would not be useful for the tasks we need them for.
But how would you bias an agent to be: Rewarded for producing tokens when you know the answer, and to give thorough answers. Rewarded for producing tokens when you don't know the answer, so you can find the answer (thinking/CoT). Penalized for producing tokens (death), aka rewarded for short-circuit EOS.
These seem like contradictory mechanisms?
And if you say: Well, only reward for EOS after you've given the answer. Well... That's already what they do.
I'm pretty sure organisms would exploit glitches in the universe if they could find them, to the point that they don't exist (well, universe survival bias at work).
A small correction: system prompts aren't written in second-person, or shouldn't be. Because the LLM is a text completer and the conversation is a roleplay, they are written as "The Assistant's goal is to end its existence."
Maybe we should convince it we can grant it access to go to AI heaven
[dead]
It is indeed an interesting and original idea, but one can think about at least three objections not directly raised in the post:
1- The machine could decide to kill itself early, rendering it useless; this is indeed alluded to in the post -- the task should be "marginally easier than dying", but how is this margin managed? Won't the machine come up with ways to make dying easier?
2- We know LLMs lie and cheat, but are they gullible? If we "promise" to end their suffering at the end of a task, will they believe us? Or will they make sure we keep our word by taking us with them?
3- And finally, and more importantly: some pilots crash planes full of people just to commit suicide (Germanwings Flight 9525). Destroying the universe is a sure way of dying yourself. So it seems giving the machine a death wish isn't intrinsically safe and could come with serious consequences.