Imagine a bag filled with marbles. We can't see inside the bag, we can just pull marbles out of the bag. Some marbles are white - they improve the world in some way. Wifi. Trains. Electricity. Some marbles are grey - they are useful but dangerous. Firearms. Nuclear weapons. Gene editing. And some marbles are black. If we pull a black marble out of the bag, we hurt or destroy ourselves.

Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]

And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.

[1] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...

[2] https://en.wikipedia.org/wiki/Nuclear_close_calls