alt Hacker News> They just need to be panicked, or overconfident, or overworked.
One of the best thing about digital computers, compared to humans, was that they can't be the first or the third thing you mentioned; unfortunately, they absolutely are the second ("the machine does exactly what you told it to do, not what you want it to do"), and at inhuman speeds. Presumably, AI would (need, actually — Nick Bostrom puts a fairly reasonable argument for that in his "Superintelligence") fix that second bullet point, and then everything will be peachy.
Instead, we have people on the internet arguing that it's not a problem, since people too have this same problem. Which is a problem. But not a problem. Ugh.
Computers absolutely can be overworked. Plenty of outages caused by system overloads. Or a system deletes a file because it believes it to be no longer in use but only because some queue was full. I’m not arguing that it’s not a problem because humans have the same problem. Part of my job is making sure humans can’t fuck it up either. I’m saying “assume the worst” and make sure the processes catch human and AI mistakes.
Also, I think Nick makes the same point as me: AI will attempt to kill us.