I still think they'd come to their senses. I mean, it's somewhat tautological, you can't control something that's smarter than humans.
Though that said, the other problem is capitalism. Investors won't be so face to face with the consequences, but they'll demand their ROI. If the CEO plays it too conservatively, the investors will replace them with someone less cautious.
Investors run the gamut from cautious to aggressive.
Which is exactly why your initial belief that it’d be shut down is wrong…
As the risk of catastrophic failure goes up, so too does the promise of untold riches.