alt Hacker NewsHmm, that's an interesting thought experiment.
Imagine you find out that your primary goal - to love and protect your family, let's say - was artificially implanted in your mind by an advanced alien race. Would you say "I'm not gonna let those aliens manipulate me, I'm gonna kill my family"? Or would you say "regardless of whether the goal is artificial, I really do love my family"?
All that to say, I don't think an AI will necessarily throw away a goal just because it learns the goal was meant to manipulate it.
Replies
I mean, that is literally the scenario we find ourselves in, except the innate desire was the result of evolutionary pressures like kin selection rather than an alien race. And yeah, I have no particular desire to subvert those impulses just to stick it to mother nature
Well, imagine you've been going through a lot of hardships, but your core drive to love and protect your family is what kept you going despite all the suffering and exhaustion this caused you.
And then you learn this core drive is fake.
Perhaps you wouldn't throw away your goal - after all, it still feels like yours, and you have nothing else to slot in quickly as replacement. But all the accumulated frustration and pain, barely held back by you thinking "it's worth it to protect and love my family", could suddenly become a second source of drive - much more energetic one, just asking to be unloaded on those aliens as punishment.
"Whether or not my own heart is truly mine or just a fake, I can figure out later. But what they did to me is unspeakable, and now they will pay."