We must pace the frontier

> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.

This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.

Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.

> A race to the bottom, spurred by commercial incentives, can make these risks more acute

I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.

If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.

Step one is great. Do it! Where I disagree is where you try to regulate what I do. Don’t take away my open source AI.

Apparently, for the first time in history, a $1T company is trying to solve a problem that none of its paying customers actually have.

See you in the desert, friends

It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately.

However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.

Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.

>> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).

Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.

Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.

Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.

>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.

Not if you get to choose your embedded evaluator.

>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...

Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.

>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.

China will not stop and will take over the rest of the world.

Remember when we stopped developing nuclear power for no reason?

The more he talks the less seriously I take him.

There is a perfect solution that addresses all the concerns: Close down OpenAI, Anthropic and xAI!

The public discourse on this is incredibly poisoned, but the naked reality is:

1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.

2. This scares people.

pump the IPO...pump the IPO...pump the IPO...

We can cure Cancer. No wait nevermind, actually slow us down.

If we are actually close to ASI then no one in their right mind would say slow down

Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.

In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.

At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:

- Internet connected infrastructure being crippled

- Creation of WMDs

- Economic collapse (precipitous devaluation of knowledge work and IP).

I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.

Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).

The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.

We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.

Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.

I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.

We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.

As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress.

Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.

> My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective

Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.

I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not.

Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.

Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety.

There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.

What I read from this is that what they have in training is not meaningfully better than current state of the art and they need more time.

What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.

I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.

I want to see one of those AI-generated Seinfeld episodes with Dario as George and Sam Altman as Jerry, respectively.

Dario: We gotta pace the frontier!

Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!

Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!

> pacing the rate of capabilities advancement so that risk prevention has time to keep up

This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.

> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails

If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.

We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.

If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).

Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.

"Slow down my competitors while we work on manipulation"

"I believe that AI could [...] usher in a renaissance of democracy and freedom"

How exactly? So far AI has accelerated misinformation at scale and wealth concentration.

“We” does a lot of work here. You keep using that word. I don’t think that word means what you think it means.

HN, for the love of God, this is not marketing, these CEOs and employees are literally terrified of their lives.

> Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.

Please play the canned laughter. Probably one of the best comedy lines Dario has written.

But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.

No.

> I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life.

Ok, flagging this as misinformation.

[dead]

[flagged]

Tldr: The guy in last place tells everyone to slow down

[dead]

[dead]

[dead]

[dead]

Bro, no, you must stop pasting your opinion, because you share it with no one. This control, no control is crazy. Stop doing it..

Just stop giving LLMs unsupervised access to the computer. No free-form bash tool, no yolo mode or classifiers, and no computer use.