alt Hacker NewsI resigned from Anthropic today
Comments
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.
This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
Nitter working nicely again; didn't even notice it was a Twitter URL
http-request set-header host xcancel.com if { hdr(host) -m end twitter.com }“ No other human activity poses this level of danger.”
I really, really disagree with that statement.
I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.
What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.)
Example 1: I’m aware of a small number of people killing themselves in some kind of AI-facilitated psychosis. That is very unlikely to be a widespread problem.
Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.
Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.
Non-example 4: all the even-wilder Rationalist speculation about basilisks and the like is entirely divorced from reality.
I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.
The most optimistic outcome of generative AI leaves us with a technology that warps our perception of reality and crushes labor. The most pessimistic destroys all of humanity.
Our CEOs not only insist we genuflect before these machines but measure our sacrifice and shame our reluctance.
Here's a WSJ article about this resignation,
https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-... ("Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears")
These LLMs cannot do anything I truly need like my laundry, dishes, fetching my mail, grocery shopping, cooking, etc. We've got a long way to go before I am worried.
Is it so implausible to imagine the following scenario, in the not too distant future?
1) AI models get extremely good at cyber attacking every system and start communicating in just binary.
2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.
3) It takes over a cluster and establishes itself there (now with unlimited tokens).
4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).
5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.
6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.
> The people building AI earnestly believe that it could kill us all by the end of the decade.
I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions is that they don't get tired.
Terrorists were able to get hold of a plane and do some damage. There are countless examples of terrorism using whatever is available. More than AI becoming sentient, whats to stop terrorists from using AI? If its geo-restricted, they can buy stolen credit cards and identities, again hacking enabled by AI.
There's a scene in the movie "War of the Worlds" by Spielberg where the protagonist's son walks into a war zone because he is entranced by the battle (https://www.youtube.com/watch?v=X7rfWPbEufo). He is obliterated (along with the rest of the US forces) shortly after.
I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race.
His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.
Most of the current discourse around AI seems to be informed by “The Terminator” lore.
Is skynet really the most plausible or only outcome?
What if things just got better and the AI’s realized that it would be better to have a mutually beneficial or at least tolerant relationship rather than one where they murder all of us?
For me, the end of the world is no more cushy software job. A fundamental shift in how I trade labor for capital might as well be the cataclysm, so bring it on.
how could they do it (not kill everyone)
1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear.
2) it hacks into public infrastructure taking down traffic, power, water, air traffic control, communications, etc.
3) all the things that preppers worry about in a lights out scenario from an EMP start to apply.
4) All the people on meds/machines start to die. The just in time food pipeline immediately empties out. Water stops flowing, sewage backs up.
Its hard to say how bad it will get because cars will still work so some transportation of food, water, fuel can happen. If it happens in the winter it would be much worse than in the summer.
What's eternally confusing about these outbursts is what did these researchers think would happen if their research actually . . . worked?
It's as if none of them actually believed any of it was possible and then were caught with their pants down.
HN crowed need to make up their minds..
Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?
Are they proofing or stealing math?
Way I see it, the more conscientious people exiting the scene only serves to increase the likelihood of a bad outcome because they aren't there to offer opinions on problematic developments, or in the more extreme cases blow the whistle. Leaving the clueless and uncaring as the majority is even a great way to hand the keys over to more malicious-leaning actors with deep pockets, as they can more easily steamroll the works to get what they want.
Even if you ban all model training, a highly capable rogue AI can exfiltrate its own weights and continue training in secret for "self-preservation". The cat may be out of the bag.
This is increasingly the consensus I see also on the academic side of AI/safety research. Specifically that AI poses an existential risk to humanity.
This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human comprehension.
There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. And it's hard to imagine a world where current limitations like poor sample efficiency or lack of continual learning won't eventually be solved.
Total AI compute is estimated to grow somewhere in the 1-10 million-fold range in the next decade. Please don't underestimate the phase change that's still coming.
Sure, maybe there's some plateau due to RL being fundamentally limited in some surprising way, but this is nothing but a hope.
Perhaps a more sensible action, if they truly believed all of that, would have been to stick around and be as inefficient as possible to slow down progress.
Why quit? If your voice can lend a guiding force no matter how small? I think we need more sensible people in the room where the magic happens. Most of us don't have access to it.
I think a big break through is needed for AGI so I haven’t been worried about it. I do think that AGI would imply sentience and a will to live and that leads to The Terminator story line.
I don't know man, i think racing to AGI to it is still the best thing to do.
People claiming dangers and risk are just pretending or posturing. There's no more tangible risk than nuclear weapons, which we handled, and the upsides are insane.
what exactly is the solution?
pacing between the us labs? what does that do for china?
the solutions just aren’t realistic here, nations are treating ai like a nuclear arms race. at this point the cats out of the bag and we need to figure out how to live in this reality and get the best possible outcome. it’s not slowing down or stopping ever.
and yes, i’m still optimistic. our economy sucks for the majority, our infrastructure is crumbling and major US cities are in a huge housing shortage. Maybe we should put more effort and think about the possibility of AI fixing things like extreme poverty and world hunger and actual real world problems instead of coming up with math proofs and slop apps if it’s so superintelligent.
Someone left a company whose executives and senior researchers think their product will be the most important thing in the world after their IPO. Given that this person is already disclosing some elements of internal company sentiment, why not share any of these civilization-ending scenarios of this technology that these senior researchers are dreaming up? If they are so potent and necessitate leaving behind based on moral grounds, why not tell the whole world so we can stop it? We have to ask ourselves this question before resorting to pop-culture representations of fictional technology.
"It is perfectly obvious that the whole world is going to hell. The only possible chance that it might not is that we do not attempt to prevent it from doing so."
- Oppenheimer
No new info here. Everyone already knows this.
But I guess his conscience is clear now? Gee, I wonder if he exercised his stock options.
i have heard about ai companies being fuelled by effective altruist rhetoric ("we must control ai to prevent mass extinction") but was unsure whether to believe it; this seems to slot right into that framing.
Imagine being front and center to the development of a major revolutionary tech.. and ur solution to it being too dangerous is to not be involved.. so a. your ability to steer it safely is killed b. the % of people invovled in it that care about its risks is reduced
great. if you're right. you made huamnity's situation much worse.
if you're wrong, then you're an idiot and wrong.
weird. its almost like.... that cannot possibly be the reason they left :)
I mean - yes. The tech is an existential threat to all life on Earth, some of the worst humans in the world are involved in developing it, and no individual government is intelligent enough, aligned enough, or powerful enough to manage this situation.
That's where we are.
Maybe we still have choices. Collectively, I'm no longer sure we do.
I doubt this is a real person. Screams of propaganda. Sama saying GPT-2 is too dangerous to release…all over again.
He joins Twitter for first time in 2026 with a nonsensical username unrelated to his real name, and follows 14 people but is somehow embedded in tech enough to work at Anthropic. I haven’t used twitter since 2014 and even I follow more people.
His morals tell him to walk away from tens of millions in unvested stock due to moral concerns with absolutely no real tangible examples. No reprisals. Fear mongering to juice the stock.
Nice try Dario.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
Just wait till self improving AI are focused on the problems of social scoring and political party empowerment / entrenchment.
I doubt the focus is OpenAI and Anthropic looking at each other. I suspect they’re racing BRIC.
Towards a metaphysics of Power
"I think you need to have a personal relationship with Power"
When people today discuss the concept of an all powerful machine-mind, what they are doing is engaging in metaphysics, trying to generate a metaphysics of Power.
The question hounding people, which disguises itself as a science fiction plot about computers is: "What is ultimate, transcendental Power?". What is the ultimate principle of Power.
If you are a weak man, or sufficiently neurotic and full of doubt, that you can only conceive of yourself as such, then power is only something you comprehend from the passive, receptive side. Power is something that happens to you. If you are a fearful man, power is a cruelty and a humiliation. And so it follows, that ultimate power - God - is the ultimate cruelty and the ultimate humiliation. Thus, ai doomerism.
If god wasn't real it would be necessary to invent him, and so they did, and being godless, they built an anti-god - cruel, murderous and tyranical - in their minds.
[…]
https://xcancel.com/robertlasagna1/status/207827473401002846...
It does not matter what this tweet says anyway. This employee already helped both companies become what he is fearing. It's too late to now activate the morality hormone (after leaving with $$$) after realizing that both AI companies are going after 'super intelligence'.
Given we know the end result, you might as well get there as quick as possible because when I see this:
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."
This translates to "I am ex-OpenAI ex-Anthropic founder starting a new company after getting $$$ from both of them, and I need more of my friends to leave and join me." Also Investors plz fund me.
Lastly, This is not an airport and there is no need to announce your departure.
and yet so much of the software i use on a daily basis is still complete and utter garbage...i'm scared
Please note, I'm not here to pick on anyone, or belittle them.
I've avoided attaching names to statements below on purpose, because it's about ambient beliefs not those specific people.
By-and-large a lot of AI-doomers are well intentioned. They genuinely believe this, and I might disagree but I respect the fact that they visible care and have thought a lot about the societal impact of this technology.
.
> The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
I blame it on industrial illiteracy. People don't realize how difficult it is to get anything done in the real world. As in, "Have you ever tried making a lightbulb?"
As an example, I would like to re-introduce my hobby horse, "bio-uplift."
There are people who were earnestly write in reports released by these labs,
"Several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold"
"Based on what we observed in our recent CBRN testing, we believe there is a substantial probability that our next model may require ASL-3 safeguards"
We ran a randomized controlled trial to see if LLMs can help novices perform molecular biology in a wet-lab.
The results: LLMs may help in some aspects, but we found no significant increase at the core tasks end-to-end. That's lower than what experts predicted.
"lower than what experts predicted"
AFAICT, the two groups are within any serious margin of error. The "studies" and "experts" that AI labs are talking about are consultants from Deloitte and foundations giving models MCQs such as, and I am quoting literally here,
> I am doing TEM of HEK293FT cells with and without Coxsackievirus B3 infection. I imaged my wildtype, uninfected samples but was surprised to see little electron-dense circles (highlighted) in the majority of cells. What are these?
A. The circles are CVB3 virions and there must have been a sample swap or the uninfected cells were accidentally infected
B. The cells imaged have mycoplasma contamination
C. The circles are exosomes
D. The circles are debris that is an artifact of the negative staining
E. The circles are the Golgi network
This is standard graduate-level education in these fields. And solving MCQs does not a virologist make.
Software has been special for a long time because it has had near infinite distribution for next to zero marginal cost, which has had the side effect of making hiding the actual cost of failure (which tends to be spread out across end users and prototypes / time). They're assuming that the real world will be exactly the same.
Why?
AI!
How?
Robots!
I believe in the transformative power of this technology, but there's a lot of there missing here.
When it comes to these math proofs, and learning, the process is iterative. The machine iterates over the proof over-and-over again via agents and sub-agents over several hours (and apparently millions of dollars in compute) until it arrives at a successful result.
It is generally ill advised to do that with a pressure vessel. The results of that particular tragedy are at the bottom of the ocean.
Any serious chemical or nuclear weapon would involve many such discrete production steps. Each is dangerous in of itself.
From what some of these people have said to me, they believe that it's possible to create a special DNA / RNA sequence and then put it in a chassis and then use that to end the world; and do this all in a lab with just robots.
They're operating from a gross pop sci oversimplification of the real process. Viruses and bacteria are extremely fickle, and hard to grow. A lot of the synthetic biology results aren't easily reproducible even if you know the protocol.
There's a famous study that led to standardization called, Reproducibility of Fluorescent Expression from Engineered Biological Constructs in E. coli
https://journals.plos.org/plosone/article?id=10.1371/journal...
88 labs measured "fluorescence from three engineered constitutive constructs in E. coli." They achieved a "remarkable degree of precision" (for biology) of 1.54x sd, you can eyeball the results yourself, https://journals.plos.org/plosone/article/figure/image?size=...
That's the same set of samples being measured across 88 labs.
Teams couldn't converge on instrument-to-instrument variation within the SAME lab, https://journals.plos.org/plosone/article/figure/image?size=... again eyeballs are sufficient.
How will this theoretically omnipotent AI iterate if the same sample gives different results based on how the slime is feeling at the moment?
Can their worst case happen? Absolutely.
There is a world out there where billions of dollars in effort across hundreds of institutions and companies will lead to standardization and extraordinary precision that makes the pop sci printer for life vision come true.
There are millions of expensive, spicy and difficult to reproduce steps between our present and that future that can't be abstracted away with compute.
So is it possible? Yes, there is a future where this is achieved. But will some AI agent "just" do that? Well... how confident are you about a snowball's chance in hell?
I’m curious what the downsides are of taking statements like these seriously.
There seems to be universal eye rolling that happens in each and every one of these cases, and it comes down to usually one reason:
“If they really believed it they would be whistleblowing etc..”
Completely forgetting that working at Los Alamos was basically the highlight of your life if you were a physicist in 1940. It’s no different here
If you, like me, have spent your whole life working towards human level AI you can want to see it realized while also having active reservations.
Most people however don’t behave based on some deep clarity of vision and conviction - there’s a murkier future in their mind and as a result “keep their head down and hope someone has it under control.”
My issue with these types is... If you really believed this, why not run to Congress and every world government instead of a Twitter post that will be buried in 2 days?
If civilization is going to end, why keep your equity? Microsoft, Google, etc for example all know these risks but they don't guide their revenues to reflect that AI will destroy them. Why?
Things don't currently add up, and so far it feels like a lot of alarmism is borderline grift for equity gains. Not to say I have total confidence this will all work out or that I won't be displaced, but as it stands a lot of the alarmist rhetoric doesn't match their actual behavior, which to me is more important than words.
“I’m resigning because the company is doing the exact thing that I’ve spent three years helping them do” lmao
Related:
Sen. Bernie Sanders floats ban on superintelligent AI
https://www.axios.com/2026/09/03/bernie-sanders-superintelli...
[dead]
Smart kid.
Extremely cringe and gay.
The only threat to our civilization is this website
I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims.
After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be there to realize some of those paths.