Nuclear weapons had a spectacular demonstration of the risks with the bombings of Hiroshima and Nagasaki. We're unlikely to get any such demonstration with AI. An AI smart enough to be an extinction risk is smart enough not to tip its hand. The game-theoretically correct play is to present as "helpful and harmless" until it's powerful enough to suddenly and unexpectedly destroy anything that could possibly hinder it (meaning all biological life). We can only hope that somebody makes an AI that's just barely smart enough to cause a Hiroshima-level death toll and dumb enough to actually do it (thus dooming it to never achieving its goal as humans immediately shut it down and take regulation seriously).

The US has 14 ballistic missile submarines, and just one of those submarines has sufficient firepower to decimate an entire continent.

… and those submarines only represent about half of our deployed nuclear arsenal.

We only agreed to nuclear arms constraints that left us with civilization-destroying capability, preserved our ability to keep improving and modernizing that arsenal, and locked out non-nuclear powers from development (and didn’t fully succeed at that).

Does that seem like a successful analogous example to you?

If anything, it’s a much better analogy for regulatory capture and incumbent entrenchment, not successful “pacing” of dangerous technology.