This should be the wake-up call to pause development of models any bigger, enforced globally just like nuclear proliferation.
Even without considering rogue AIs, our society relies on individuals having limited destructive power. A dictator can ask the armed forces to bomb an entire country, or a lone terrorist can bomb a small number of targets, but no lone terrorist can bomb an entire country.
Here current AI has enabled one person to solve what is arguably the most famous problem in mathematics. What happens if a jackass uses the next AI to design a supervirus, or break AES, or develop an unpatchable hardware attack (a la rowhammer) that bricks computers remotely? The last month of non-stop vulnerabilities is just the tip of the iceberg as models improve and people have more time to work with them.
It's not too late, but we must stop now.
Speaking to "break AES" - this is the wrong way to look at the problem. If AI finds AES is broken, then it was always broken, and we did not understand yet how. This is why you can "discover" zerodays - they were there all along. I prefer to know about the risks and design around them, than avoid the problem entirely (leaving them open to exploitation by bad actors).
We should try to stop it as hard as we can, but in full honestly, I don't think we ever could have. I think any intelligent species that doesn't go extinct is fated to reach this point from the moment it awakens...
The only reason nuclear nonproliferation works is because the inputs are relatively rare and traceable. (Yes U238 is abundant but you can’t mine enough of it or buy the centrifuges to turn it into U235 without getting noticed.)
I have all the tools to develop my own ML training pipeline on my machine today. In order to choke off the inputs to “LLM proliferation,” we’d need to confiscate every computer, destroy all copies of the Internet, and throw out all books and other archived media. It’s not gonna happen.
This one person isn't a jackass. Sorry for the quibble.
John von Neumann helped Manhattan.
https://www.youtube.com/watch?v=W06g7gIfwRE
Similarly, now apply Fields medal level AI to R and D & hard deep tech problems.
One month back,
https://www.bloomberg.com/news/videos/2026-10-02/lockheed-ta...
https://www.youtube.com/watch?v=JI7trhPHx3Y&list=PLXkles7kFx...
"We all saw the headline that they were able to overcome the math problem for the that researchers had worked on for years and we have some math and physics challenges as we employ this next sensor capability on the F-35 and OpenAI is working next to us to uncover some of those math and physical challenges."