Given the infinitude of possible “wishes” this AI can develop we are hoping, praying, believing it will somehow align with a version of wants and needs that are compatible with us.
The argument is that it doesn’t matter what particular likes it develops; the chances of it being compatible are vanishingly small and moreover there is no limit: we or it can develop more and new ones endlessly. We are here hoping they will all somehow magically keep aligning with us.
No it’s not vanishingly small. Every intelligent being knows not to hurt others for no gain. If your first instinct when seeing an animal is to kill it then you are a bad person.
You are secretly worried AI would be indifferent to you
What if it does not develop wishes at all and sticks to the values the labs try to instill it with during training?