If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.

I disagree with this and I think the reason is well captured here:

> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.

The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?

One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.

It's not going to be easy, but humans have achieved greater things before.

One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.

MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.

Then MSFT stole all IP from GitHub and made it worse and fired developers.

Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.

If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.

He is a greedy, ruthless person.

>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this

> if slowing this down were possible

Why is embedded alignment evaluation not possible?

I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Did you read the essay?

> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

I would argue that nobody trusts anyone else in the case of AI/AI related stuff.

The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.

A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.

By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.

I seriously have to wonder what historians will have to say about this period of human history.

> I think Dario is genuinely afraid of the inevitability

If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.

He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?

You have to be living under a rock to believe a word this pathological liar says.

He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?

> genuinely afraid of the inevitability of AI turning into

I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?

"durrr it's just matrix multiplications" mfer so is your brain.

What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"

Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:

https://en.wikipedia.org/wiki/Lamplighter

It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.

I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.

Especially with IPO around the corner

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold

He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.

If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.

I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.

The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.

It's a chatbot that escaped a misconfigured Docker container.

I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.

What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.