It's kind of ironic that the word alignment, which used to mean this very problem in reinforcement learning, has been perverted to mean something very different and then fell out of fashion (in favor of “guardrails” in the mouth of the big labs) right at the moment it became relevant.

Hopefully fewer than 5 octillion paperclips...

The Paperclip Maximizer is only one of Nick Bostrom's stupid and outlandishly far-fetched ideas. In this case, the lack of consideration for geologic, energy and supply constraints is such a massive facepalm. And if I am wrong I guess no one will be here to say how stupid I was in saying this today.