Ages ago, a failing physics student decided that he would make creating a nuclear bomb his PhD dissertation, operating on the theory that it would have sufficient notoriety that it would make getting a job after graduation easier, and that it would also serve as a cautionary tale on the problems on non-proliferation --- one of the things which enabled this was that his thesis advisor had worked on nuclear weapons and still had the relevant clearances.

The student thought to test the validity of information by showing his teacher two different conclusions, one which he was confident was correct, the other he was certain was wrong, expecting a different reaction for each --- instead, both were acted upon in the same way, the subject was changed and no explanation was forthcoming.

He was eventually successful, earned an A and his Doctorate, and his thesis was classified.

>Of course. "tried to" being key words in the comment.

Fair enough, I misread your original comment.

The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.

I think you're picking the wrong example. If I had some sticks, a bit of mud and a few leaves, whether or not I had Claude wouldn't make a difference to my ability to make a nuclear weapon. There are probably better examples of ways where unmediated AI might facilitate something horrible, although probably on a smaller scale.

> A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you."

That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome?

There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace?

But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration.

I'm sure China et al are already doing this.

For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.

He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.