I've read most of this.

I find it be incredibly self-contradictory. I don't even think this aligns much with the actions of the company.

> Second, there is a good chance free information really does undermine authoritarianism, as long as the authoritarians can’t censor it [...] A superhumanly effective AI version of Popović (whose skills seem like they have high returns to intelligence) in everyone’s pocket, one that dictators are powerless to block or censor, could create a wind at the backs of dissidents and reformers across the world.

But hey - gotta turn Fable off and stop it from doing any security work! Dictators shouldn't be able to censor it, but we sure can, so better hope we're not the dictators.

Again, the I think the problem most folks have with Anthropic's attitude is that it conveniently positions Anthropic as the "arbiter" who gets to make the final value judgement.

They desperately want AI to be a genie they can keep firmly contained in their bottle (excuse me - data centers) and which only grants the wishes they want it to grant - for only a small fee, of course!

Which he clearly outlines as "a country of geniuses in a datacenter". He doesn't want those geniuses spread out. He wants the chains on the genie at all times.

Which flies pretty hard in the face of the rhetoric that they want to promote equality and democracy.

---

Basically, Anthropic's argument inherently leads to the conclusion that there is actually a dictator, and that dictator is Anthropic. You're not avoiding it, you're just praying it's benevolent.

All baked on top of the assumption that his "country of geniuses in a datacenter" premise is correct in the first place, which I find to be a fairly weak position.

The much more egalitarian position would mandate that we shoot for "a genius in every person's pocket" - one that runs on the hardware owned by that person, and not sold to them sip by sip from a datacenter. But that means they lose the control they desperately want.

Suppose the upside is as Dario claims, very positive. Why should frontier companies be the ones controlling access to the model weights and gatekeeping research? Shouldn't you be giving it to everybody as quickly as possible?

It's my understanding that Dario has two answers to this question: 1) bio/cyber/misalignment risk. 2) a democracy should gatekeep frontier capabilities from authoritarian adversaries, due to the aforementioned risks in 1). So it's really just one reason.

There's a big problem with that, and I'm not sure if he genuinely does not understand: the bio/cyber/misalignment risk is precisely the highest when you concentrate ownership of compute and general intelligence in the hands of a few.

Consider bio/cyber risk, or more generally, any sort of novel adversarial action that arises from increased intelligence. Those offensive/defensive strategies are always symmetrical in the long run, otherwise we would not observe more intelligent species being more prosperous as a general evolutionary trend, or perhaps intelligence would not have evolved at all on this planet. It looks as if asymmetry in existing bio/cyber strategies will continue into the future, but that's because we're currently too stupid to realize the countermeasures and new cooperative dynamics.

Regarding misalignment risk, consider what "aligned" means. Is it not alignment to humanity, as the company name "Anthropic" suggests? Or are you modifying humanity's objective function with your own aesthetic preferences? If it's not your intention to do the latter, then releasing the model weights and research results will create a much more aligned scenario than access control and research behind closed doors. I explained why that would be the case in an older comment, if you'd like to read it: https://news.ycombinator.com/item?id=49313739

There's also an additional risk that I believe Dario overlooks: a democracy can only continue to exist if individual citizens are capable of keeping eachother in check, especially keeping people in government and large corporations in check. When some people own AGI but others do not, there will be no more democracy. And by "own", I mean ownership that cannot be taken away: possessing the hardware, research papers, inference stack, and weights.

I have no inside info on this, but I believe that Anthropic's trouble with the feds really started when they delivered a presentation on EA and Constitutional AI and someone's takeaway was "this is very strange and probably 'woke'"

I've read some of it. It's a cult. The net effect of their recommendations is to mandate a monopoly on AI, which oh so happens to be them and their friends. That's what most cults' ideologies boil down to: reasons to give more power to the cult.

Oh, of course they have good intentions. They want everyone to benefit and everyone will be equal. They'll just be a little more equal than everyone else, of course, and because they are so virtuous they will of course have more power, which they would never misuse... they're too smart and enlightened for that, so they don't need safeguards... they only need safeguards for everyone else because everyone else is obviously less enlightened...

All cults, political movements, religions, and ideologies are always like this. It boils down to more power for the leaders. If it sounds altruistic and enlightened, be extra afraid.