Our position on open-weights models

What would it mean to crack down on distillation? I could think of a few possibilities:

1. Using political pressure to target companies that are accused of doing it.

2. Attempting to impose criminal penalties on individuals associated with the action.

3. Having the US government attempt to use its capabilities to stop it.

None of these seem particularly likely to succeed.

He wants to restrict China from access to chips. All that will do will accelerate their development of competing, sovereign hardware.

As an example take North Korea. Sanctions didn't stop them from developing nukes and delivery systems.

Dario has like three 'paranoias' / strong-motivating-concerns

1) LLMs turning into Skynet

2) China as geopolitical competitor

3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)

So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models

It's absolutely reasonable to have safeguards on sufficiently dangerous models being released - if you disagree, can you explain your perspective?

I think it's wildly irresponsible to release models that are extremely capable at things like bio-weapons. Do you really think information anarchy is the answer?

The problem with open models compared to closed models is not about protecting profit - it's about protecting capability. Any open model can be retrained or fine-tuned for anything. There's no such thing as an open model that is both capable _and_ permanently safe when it comes to certain dangerous topics. It's not possible to prevent 'uncensoring' a model.

He says that using the set of questions and answers from one model to train another model (deatilation) is cheaper than training the model without those datasets.

But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.

In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.

> Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models

I don’t think this is open or closed; this is aligned and unaligned. I bet Grok would be as open as any open weight models to answering questions.

Are guard rails meaningful if they can be removed from the weights? Can America even prevent the release and proliferation of these models?

It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.

Dario thinks of policy as if the Berlin Wall fell yesterday, he is so detached from the reality of the world.

The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.

I'm no longer supporting this vendor, for personal or business

"No guys, Anthropic actually loves open weight models!" Yeah, and Microsoft famously loves Linux. Sure.

http://www.omgubuntu.co.uk/wp-content/uploads/2018/04/micros...

The core concerns stated as use of AI in drones and surveillance, by China. And what does USA government do with AI? Drawing pictures of flowers and writing novels?

Hmmm, how could this post be possibly a good thing for Anthropic?

It just also happens that open weight models are a massive financial threat to the existence of Anthropic as a company… so the might just have something to do with this position.

Conclusion: open weights models are good for Anthropic because they shows the so called threat that will make congress allow pouring money in Anthropic for national security & AI arms race

Americans don’t get to lecture the world on authoritarianism anymore

I welcome any and all competition. But they first need to be able to run the obstacle course I’m setting up with Uncle Sam.

If your concern is that China will develop models that are significantly more powerful than those of the US, why would you care so much about distillation? It seems like distillation is a way to catch up on capabilities, but not so much a way to jump ahead in capabilities.

Meanwhile Chinese chip-makers are chip-making. If you believe the threat of these open-weight AI is existential, how long do you think these bans (that only work for hardware) will work in your favor?

Clearly anthropic is misaligned, how can they claim to be able to align AI when they themselves are misaligned from humanity.

"Open-weights models that don’t have dangerous capabilities are a public good"

Read between the lines folks. Anthropic deems every model that has frontier capabilities as "dangerous", and thus they are against them. We all know that "dangerous" simply means "whatever model hurts our bottom line."

More dishonest framing from the company that constantly lies to everyone. No surprises here.

I found lots of comments here suffer from proximity bias.

Anthropic anti-open-model stance does not mean China is not a threat.

Banning exports to China is merely a narrative device; the real intention is to sideline intelligent technologies.

Such an incline for centralization makes unsustainable and/or overleveraged governance systems.

Statement is a whole lot of nothing, as expected, but I also don’t know what people are expecting from these guys. That Dario will have a sudden change of heart and publish weights of all his models, flushing $1T down the drain?

oh, it‘s the CEO of a well known AI company educating us all – and all he wants is us to hear his hunch on open weight models?! Amazing, please help us understand the situation a bit better, thanks

"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category." Translation: If it's so strong that it threatens my business, ban it.

"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.

"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.

I like the idea that Mr. Amodei holds these beliefs because he wants to safeguard democracy as much as the next guy.

I just don’t find it believable.

At least try to release one open weight before you say anything, or all of that only sound hypocrite at best.

Imagine chinese regime getting access to powerful models who bombed school students in Iran amusing AI !

> We should not sell powerful chips or chipmaking equipment to China

And accelerate their development of independent chip making technologies even more…

Well their valuation is going down the drain so no wonder they don't like it. The cherry on top will be China developing their own chips and chip making tech.

The consent factory has never been more productive.

Anthropic will continue being a victim of their own naive positions on AI safety. They keep dancing around it but their communication is essentially pro-regulation if you read between the lines.

Seems pretty reasonable.

I agree with Dario Amodei. Every word makes sense.

distillation: Pirates people’s lifetime of copyrighted work, makes billions selling access to it through APIs, then tells us we can only use it in ways they approve.

"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."

Hmmmm.

Hey, no fair stealing the value thay we already stole fair and square!

> Anthropic has never advocated for a ban on open-weights models.

If you wonder why this is written as an opening to a list of reasons that advocate for banning the open weight models, it’s because

Perhaps, the general public needs to formulate its position on Anthropic models...

This seems hypocritical, out of touch, and hollow. "Safety testing" is just a hair away from censorship when it comes to government control.

When they use ill gotten data for training their models the world's smallest violin plays when they complain about distillation attacks.

> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.

Welcome to bizarro world!

Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.

Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.

I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.

> Anthropic has never advocated for a ban on open-weights models.

Shut the fuck up

I wonder if publishing these documents is not just a public stunt, but heavily integrated with Anthropic's business storategy to maximize operational efficiency. Companies often have several internal documents for a single policy like "position on open-weight models", one for public (like this), others for the legal team, the lobbyist, the developers, the investors, etc. The differences and nuances of those manuals can be very huge and are necessary to maximize the goal from each branch, but often a cause of headaches like bureaucracy, communication friction, outdated information, etc.

A single canonical official document can make it very simple. Even though each department cannot achieve the maximum gain from nuanced documents, keeping operational context as simple as possible may really improve LLM driven operations to move faster and cut cost.

If "publishing pleasant positions and actually following them in general" becomes a good business storategy in LLM driven society, it can be one of very few good outcomes from this dystopian AI craze.

A sober look at actual rogue nations and their use of AI shouldn’t have us fretting over that particular hemisphere

he's asking for the most powerful models to be in the hands of the few while the majority will be at their mercy.

> Open-weights models that don’t have dangerous capabilities are a public good

Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.

The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.

I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.

Their position seems fair to me - sure it does help them as well, but I don't necessarily disagree with the risk they are laying out.