alt Hacker News> It is also why the doomer argument is so ridiculous. It is a type of superstition that is confusing a real danger. We don't need to "align" the nuclear reactor so it doesn't get angry and meltdown. It is just a machine that we need to make sure doesn't malfunction in a way we don't want it to.
It's not ridiculous, unless you misinterpret it like you did here, and link it to consciousness.
Doomer argument is just that optimizers gonna optimize, and our own intelligence is just an optimization algorithm too, albeit highly general one -- and once we're building optimizers with comparative generality, we're facing possibility of one that out-thinks us and we can't control it anymore. The "alignment problem" is about how to ensure such sufficiently-smart optimizers end up optimizing for the same things we are, because any significant deviation here ends up in doom for us.
Consciousness never even enters the picture here. And it's not a given than it should.
(Incidentally, while hard sci-fi and not a paper, but also entirely unrelated to AI X-risk crowd, the books Blindsight and Echopraxia sketch a pretty convincing argument that consciousness may be an evolutionary disadvantage and it's not only not necessary, but may be holding us back.)
> Doomer argument is just that optimizers gonna optimize
Right. I don't quite subscribe to the strong doomer argument, but take water:
Water flows downhill. It's not conscious. I don't think it's anthropomorphism to say that water "seeks" the lowest point on the landscape. Often that seeking process is mostly harmless, sometimes it's devastating, and that's where you want levees and dams.
Now expand the same mathematical intuition out to other lowest-point-seekers in a high dimensional space. You don't even need to postulate a mind per se to see how optimizers can cause quite some mischief.
Meanwhile, Kids and Kittens are good examples of this behavior at the intelligent end of the spectrum: Given time, they'll find a way to get through any gap. They don't need to be as smart as the adults, the adults just need to turn their back for one minute too many. O:-)