logoalt Hacker News

mindslight • today at 5:02 PM • 0 replies • view on HN

One difficulty is that "accounting" is not a neutral thing. Rather, once something becomes legible to the paperclip maximizers, it is something that can be optimized away in favor of more paperclips. And sure, we can make the case to the paperclip maximizers that perhaps their accounting isn't correct, and that there could actually be more paperclips in the long run by not optimizing away this one particular thing. But that is still a Faustian bargain, in that the paperclip maximizers may very well just find a different way of creating paperclips that simply sidesteps the need for humans in the first place - rendering your last-ditch attempt at appealing to their interests irrelevant.