> Kinda like nuclear bombs or similar.

...and AI progress is actually a much more difficult thing to control than nuclear proliferation...

Unlike nuclear bombs there is no rare difficult-to-process ingredient which you can practically limit the supply of, unless you are willing to retroactively outlaw computing devices broadly (after we have spent decades weaving them into the fabric of most people's day to day lives).

And even if you were willing to try that, how do you get global consensus retroactively? It is an impossible situation.

None of what I've written here should be seen as a defense of the people running large AI labs, I don't believe any of them are forces for positive good in the world, but it is just a fact that if they don't "do it" someone else will.