I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI.

1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.

2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.

So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.