An LLM can't help you build a nuke. LLMs can't paper over hard engineering challenges like "You need to reinvent the centrifugal separator industry" and "rocket engineering is pretty difficult" and "you need really really good data to simulate things before you can actually build an implosion device" and less engineering challenges like "The CIA keeps murdering anyone who works on this"

There's never been an information access problem with nuclear weapons. The information was so public that most physicist realized everyone was building bombs from the sudden lack of published papers on how you could build bombs.

It's just silly how people seem to believe that if you just have a page worth of text saying something, you can suddenly do that thing.

Even a magical superintelligent AI could not do these things. It's going to tell you the exact same thing that has been known for nearly a century now:

"Hire 100 engineers and a couple physicists and 1000 very skilled machinists and laborers, and spend a billion dollars on plumbing, and then do that a second time if you need to deliver the device using a missile"

Meanwhile a dirty bomb is trivial.

These things don't happen because people who want to cause mass harm are generally poorly resourced.