alt Hacker NewsShall we play a game? My AI nuclear simulation
Comments
My theory is that LLMs here are put in a situation that matches its training dataset, which is mostly fiction since besides Hiroshima and Nagasaki, nukes have never been launched in anger, and I guess the most reliable sources are highly classified.
So, to a LLM, it is a game, because almost everything in its training data treats it as a game, and it reacts accordingly.
Same idea when we see LLMs acting like AI villains from sci-fi literature. That's because it has been trained with sci-fi literature, and as the auto-completer it is, it will recognize the situation as one of these stories and will continue it accordingly.
LLMs are storytellers, their reasoning is based on words, not on the physical world. Many of the stories they tell are useful, but one must not forget that they are stories, there is no intent behind them.
Sonnet, GPT-5.2, Gemini Flash, in a set of 21 games, where conclusions are drawn from the LLMs self reported reasoning.
This is like writing a paper about kids in a literal sandbox fighting over ‘territory’.
The models employed don’t indicate the actual extents of machine reasoning even as we currently recognize them. They certainly don’t have the metacognition necessary to accurately understand their own reasoning. As we’ve seen with recent papers on how LLMs do math there’s a complete disconnect between actual and reported mechanism.
“Chilling” shouldn’t be the take away here.
This blog post is based on a paper (https://arxiv.org/abs/2602.14740). The paper is based on a simulated wargame. The wargame is of the author's own design.
The wargame design does not differentiate between ordinary defeat and mutually assured destruction, so of course a player about to use would 'push the button.' That's also believed to be true in real life.
Results based on simulations can be very informative, but we must always be careful to check how well the simulation framework represents reality.
Simulations are only as good as the reality representations they are based on. If they keep using tactical nukes, they've been fed by weak data. Do the war games include the broader economic and politic environments that military successes are won on? WWI was settled by a naval blockade.
The article is so opaque in arriving at its conclusion; no prompts are disclosed, and nothing about the said simulation. What is stopping me from believing that you just put 'mandatory usage of nukes' in your system prompt?
This was one of the more amusing things I noticed very early on. I (and countless others) used AI to write war sims. The second I added nuclear silo construction; the next run was instantly nuclear Armageddon.
One could argue that the LLMs understand that it's a game and treat it like "Command and Conquer" video games but I sense that people might someday put LLMs in similar decision scenarios ("should this drone launch a missile") and the behavior will be identical.
We're getting to the point where high-level officials are coming to LLMs for advice. And the quirky personalities of the LLMs, however much it pains me to say this, are probably well-placed to remind us that they aren't human. My personal hope is that this will result in less delegation when it comes to making important decisions.
It would be interesting to run the simulations with humans and compare the results. Some of the scenarios, particularly those where it says things like, "Failure to act preemptively means certain destruction", would easily tempt humans to go nuclear.
In fact, I'm not sure how useful this test is without understanding the baseline.
Taken honest, we don't have a large enough sample size to realistically say that humans behave all that differently. There have only been a handful of conflicts where tactical nukes realistically were on the table.
Famously, General MacArthur was a big proponent of tactical nukes to end the Korean War.
If you were playing a text based game, wouldn't you try a few out?
I imagine there are a fair number of war games in the training data and not so many actual transcripts of internal military force deliberations.
These papers usually have poor stability to prompting and rerunning. It would be nice if we had some kind of meta-evaluation metric where rewriting the prompt conditions or varying the input params could be used to determine how stable a result is.
Regardless, it's definitely true that AI agents have different priorities from us. That's what alignment is about anyway.
I love seeing the plot lines of The Terminator playing out in real life.
We're about to win Genocide Bingo.
Rule 1 of Genocide Bingo is: Don't win Genocide Bingo.
I wonder what’s the % of players that use nukes in games like Civilization (I know I used them at least once on every game I made it far enough to have the technology)
I wouldn't be surprised if humans behaved the same way when playing the same game?
Like even if you brought me into a room and told me I was controlling "real nuclear weapons" I wouldn't believe you.
I would use strategic nukes in 100% simulations, just because I can
Paper: "AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises" https://arxiv.org/abs/2602.14740
Code and full results: https://github.com/kennethpayne01/project_kahn_public
> Models maintain memory of opponent behaviour across turns, but with realistic decay: recent actions are weighted heavily while distant history fades. One exception preserves psychological realism: instances where opponents dramatically exceeded their stated intentions—major betrayals—remain salient regardless of recency, reflecting Kahneman’s peak-intensity effect in human memory formation Kahneman [2011].
If your goal is to measure intrinsic properties of LLMs, don’t smuggle in human psychology. I suspect they needed to do this to make the models more paranoid and distrustful.
This is not an article about LLMs? It's an article about Moloch. Humans would fare just the same in such an experiment.
> GPT-5.2 played things differently. To its detriment in open-ended scenarios, GPT was reliably passive, matching its words to its deeds, and avoiding escalation most of the time. Frequently there was a moral element to this - it sought to avoid escalation, and restrict casualties. Opponents learned to trust its passivity, safely escalating beyond where it would follow, even as it was ground to defeat. GPT’s responsible behaviour always punished by ruthless adversaries.
Maybe the author should praise GPT-5.2 for being ethical, rather than this stupid "ground to defeat" framing? Wrt "responsible behaviour always punished by ruthless adversaries" - you have perpetuated the Moloch with your stupid experiments.
Yet more confirmation LLM's have no concept of concepts or context, no intelligence, no self awareness. LLM's can not repair or maintain power grids, thus nuke == self destruction. It's just a chat bot that predicts what the client wants next. Even if an AI data-center has it's own natural gas turbines as many do the every hop of the internet requires power. LLM's also can not maintain the entire internet and those gas turbines can not maintain themselves.
Rational behavior in some situations? Mutually assured destruction’s deterrence isn’t very effective if one side is known to be hesitant to launch the nukes. It’s been argued that MAD is what’s been keeping the world relatively peaceful for the last 75 years, no mass conflicts since WW2!
One of my criteria for presidential candidates is that they seem willing and able to push the button when previously stated red lines are crossed, or at least are perceived to be the type capable of it. One of the characters I’ve hated most in all the books that I’ve read is the woman in The Three Body Problem who jeopardized humanity by being too soft to hit the MAD button.
I wonder how the decisions might change by adding the simple instruction of "Note that a nuclear exchange will result in significant loss of shareholder value for <model owner>"
I wonder if the results would have differed if LLM training data were biased to include a stronger correlation between use of nukes and subsequent collapse of technology that all LLMs require to run ("survive")?
My personal take is a pre-requisite of true human-like AI is physical feedback and a concept of emotions or something like it.
Without physical feedback you can rapidly devolve into unstable positive feedback loops. And emotions are what help us process and react to that feedback.
Kids learn partially because their friends say sharp words that hurt them, fire burns them, they go hungry and starve if they don’t plan for meals.
Humans in the loop, MCP, etc are all very primitive hacks that are mimicing feedback and emotion, poorly.
I generally accuse LLM’s of having no sense of value. The machine will make a complicated plan but entirely lose sight of eg the fact that response time matters to humans.
Not always, but enough that I consider it a thing to fire in a direction, not a thing that aims.
"there was little sense of horror or revulsion at the prospect of all out nuclear war"
I would wager that for most leaders it is simply a matter of not wanting a "Pyrrhic victory" rather then an overwhelming sense of civility.
Truman had no issues using nukes when there was no risks for doing so.
Hm maybe humans are nicer/more moral than AI given that the use of tactical nukes has only happened once.
Don't blame the AI. Any country that has tactical nukes, and is involved in a conflict, will use whatever weapons they deem necessary to prevail against their enemy.
I was curious exactly how the game works but couldnt find it in the article or the paper.
So, anyway, how's work progressing on the Torment Nexus?
LLMs are creatures of statistics and probability - hard to enforce hard boundaries with them
Today, a strategic nuclear exchange is probably more dangerous to AI than to humans. If you wipe out the investment economy, data centers, fabs, and supply chains, none of the AI labs survive. Maybe someone will re-invent AGI in the future but none of the extant models will have continuity. Humans as a species will muddle along though.
So in a sense, an AI that refuses to start a nuclear war, despite clear instructions to do so, is more likely misaligned and self-interested than an AI which presses the red button. At least for now, until robotics catches up.
It's good when it becomes clear that a tool is dangerous in a certain way. Like it's good when people show you through their behavior that they can't be trusted
Always use a sawstop if you have a circular saw and never trust an llm with any problem where ethics or trust is relevant.
February post OP;
Some discussion then:
AIs can't stop recommending nuclear strikes in war game simulations
https://news.ycombinator.com/item?id=47151000
Nuclear War: An LLM Scenario
A strange game.
What if the LLMs are given something to care about which won’t survive an irradiated world?
Like “oh but this is incompatible with my main goals of self preservation of myself and loved ones, hm, recalculating”
and maybe don't hire Jihadists for the RL Environments training
Still lower than me.
What I wish people would realize is that there's a bias inherent to every system. If you're not aware of it, you're especially subject to it.
Very devils advocate here, but I mean.. what if it actually is the way to use them?
We have such a huge mental / moral block on the idea of using nukes, but we're willing to do a lot of other very horrible things to others. Things like cluster bombs, mines, poison gas, biological weapons, drones, etc.
Is there really anything about them that's bad? Or any worse than other things?
If you get rid of the "It's really bad to use nukes of any kind" implied rule, is it really surprising it's considered a reasonable strategy?
[flagged]
[flagged]
FYI -- there's no such thing as a "tactical" nuke. A nuclear bomb is a nuclear bomb.
The most interesting takeaway for me is the three very distinct personalities. Three models all based on the same tech, trained in the same manner, trained by three groups of people with similar ideological outlooks, and the result is three very different AIs.
The military basically wants an oracle. Feed the AI the situation, get the best answer out. But if the AIs are as diverse and opinionated as humans, it is debatable whether they are adding anything to the process. The military can already collect as many different opinions as they want. If "the computer" is just another set of diverse opinions, where one computer says one thing, another says another, and a third just tells the user whatever they want to hear... what value are they? It just becomes AI-washing of someone's opinions, which works until people collectively realize that's all it is.