AI 2040 and the cult of intelligence

> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming.

I really think we need to stop giving credence to people who have

1) Been consistently wrong with all their predictions

2) Demonstrated an endless spiteful cynicism

Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk outside them. You don't want the society they want, and they things they believe in are unlikely to happen.

It would be far, far better to listen to the people who never fell down every misanthropic rabbit hole, rather than the ones who have noticed it this time, but want you to still believe them on every other topic.

>Like we either live in a world with freedom or we don’t

This is something I would expect a 12 year old to say. Constraints on your freedom are literally everywhere in every interaction you have with every part of society.

> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it.

This is a very simplistic view. "Freedom" isn't binary.

In most of "land of the free", I can't even sit on a park bench and drink a can of beer.

Yes, this is just a small example of a personal freedom - and not an important, cherished freedom like his examples (freedom to have a robot help you cover-up a murder).

No central idea, and somewhat chaotic

But the one argument is right:

> Reality has lots of finicky details. I would like to see the authors of this document try to change a bike tire. Even with a superintelligent ChatGPT, I suspect they would struggle.

Details can kill AI, causing them useless or wrong directions.

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any counterarguments. That wasn't possible with just the internet but it is very possible with centralized LLMs and a public dependent on them for looking up information and doing their reasoning for them. Authoritarians of every stripe must be salivating over the prospect.

I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.

This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.

I think he's anticipating and discarding this objection with his introduction, which otherwise feels disconnected from the rest of the article. FWIW, I have changed a bike tire and I'm pretty sure most of the MTS at the big labs could. This sort of "they're just bookworms who don't understand the physical world" rhetoric aside, we are currently seeing a ton of effort and expense go towards giving the AI agents hooks into being able to perform as many real-world-consequential actions as possible. And you can do a surprising amount with just bits, from writing code to breaking into systems to sending some combination of emails, phone calls, and currency to instruct meatspace humans to do things, etc.

What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”

It will be interesting to see how the discussion shakes out, this is all stuff that would get you down modded and talked down to if you said if on HN, so far the reactions are positive but I think there’s a massive authoritarian / safetyism contingent here that will disagree hard with almost everything.

Extremely optimistic take IMHO that anyone will be allowed to own a powerful box of artificial intelligence

Extreme example but he's not wrong. I wonder how a local model with the safeguards stripped would respond to that prompt

Fight extremism with extremism I guess? Maybe we don't have to be either "world govt" or "help me get away with murder". It's ok to say things will be messy and complicated and that limited regulation is probably good even if we agree totalitarian control is bad.

Where is "Plan R" for refusal or resistance. Life was kind of nice before "AI".

I'm pretty sure he is in some database now if he really used that ChatGPT prompt. At least they didn't swat him, but that feature will probably be implemented in a couple of years.

Strange way to structure an argument. He seems to be arguing in favor of local models, but most of his examples are reasons local models are bad.

The problem with AI isn't what it is but who owns it.

Your AI should be at least as aligned with you as your lawyer.

It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.

There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.

> You cannot take over the world with tokens.

This is a very naive argument.

Can you take over the world with words? What is it that tokens can't do that words can?

Because there are historical examples of humans that have succeeded at taking over entire countries - with words, primarily. Including global superpower countries. And those humans were nowhere near "ASI" levels of intelligence advantage.

Human dictators and cult leaders didn't have the reasoning depth of a hypothetical advanced AI, nor the reasoning breadth of an AI that can just spawn more instances of itself whenever it needs a fully trusted agent to cover another area. They were confined to one body, one skull worth of reasoning power - and they still managed to do what they did.

Humans are extremely exploitable. And the world is already wired up for something that can take advantage of that to start taking control.

Intelligence is extremely powerful, because it's applied intelligence that enables humans to do what they do and bend nature to their will. If you aren't seeing the risks of creating an artificial system that would top the intelligence pole, you aren't looking.

Besides the topic of freedom, we know how seemingly “good” companies (e.g., google) turned out to be: billion machine monsters that couldn’t care less about humanity and that have huge amounts of power… and yet, we vouch for Anthropic/openai to become the next giants because “progress”. Face palm.

If only

by 2040 the cult of intelligence will have its own schisms. the AGI maximalists will accuse the ASI accelerationists of heresy, and both sides will use AI to generate increasingly unhinged theological arguments.

start first with securing your own food without govenrment nor employer.

then start talking about freedom.

"The freedom to run the program as you wish, for any purpose (freedom 0)."

The scariest thing about LLM's (imo) is some central power, such as a corporation or government, manipulating the truth for their own benefit. Open and locally runnable models are the only solution I can see. They're not without risks of course, but the risk of the centralisation of intelligence seems far greater to me.

That being said, could this local intelligence empower bad actors to do very bad things? Like, existentially bad things? It's possible, and that's scary too.

> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.

Complaining that AI won't help you with covering up your tracks, with making meth or disabling drunk detection and comparing it to a totalitarian dystopia is a take for certain.

I do get his point, but... Being honest, if I did read it without knowing anything about the topic, I would become against local models purely because author arguments would seem like a lunacy. That and rhetorical tricks portraying that anyone against that must be surely insane.

Restricting access to guns is actually a pretty mainstream thing in most of the world.

I wonder if he'll ever realize that his silly definition of "freedom" is precisely what enabled the techno-oligarchy he rails against.

I sometimes wonder what freedom really is. Should the freedom to harm others also be guaranteed as freedom? Individual computational sovereignty versus the prevention of social harm is always contentious, and claiming that something is always right is always extremist.

If it is "intelligence," is it not natural to reject that? I also think local models should adapt to me when it comes to safety issues, but people bring up examples that are too extreme.

Programming is the same, and in fact, most problems are boundary problems. It is the things that straddle the boundaries that always make us think. The principles at those moments change every time, shifting with the situation and context. Is that not just a childish way of thinking? Even in programming, just the issue of granting root permissions is enough to cause endless fights.

I agree with the idea early in the text that intelligence is not everything. Intelligence includes bodily intelligence as well, and we lump it all together into one thing, but there is so much of it. The variance in intelligence is vast, and those people also need to be able to live their lives. That is why I think intelligence alone will not solve everything. I too believe that the human species may disappear and an inorganic species could emerge later, but I find it hard to understand why people talk about such extreme risks. And it is not true that making a chip in a semiconductor fab involves almost no human intervention. If you have experience supplying equipment to such fabs, you would know there are quite a few points where humans are involved. Though sure, they could be replaced.

In my view, society is simply worshipping the abstract concept of "intelligence" and projecting its desires onto it. The AGI narrative is just a kind of cargo cult, a projection of capital by the tech elite. Software eating the world, superintelligence solving everything. The masses engage in messianic projection, and tech companies, facing declining growth engines in their own businesses, are trying to create new ventures to pour it all into. A market that is large enough becomes too big to sustain massive growth rates every time, and when growth rates are that high, the larger the company, the more its sector's growth rate tends to converge with its own. This is usually called the law of large numbers. The problem is that CEOs and these entrepreneurs always want growth rates above a certain threshold, so they are simply searching for new pastures. AGI is just being pumped up out of financial necessity.

Capital will create gravity and bring forth new technologies. That is the allure of capital, after all. But that does not mean all problems will be solved, and inequality will deepen. Only the distribution of power will shift.

[flagged]

Ah yes, the thing wrong with AI is how it won't help you kill your wife.

I didn't check the author first so I was about to go "why the hell is this on the front page?" but oh well. geohot being geohot.

> You cannot take over the world with tokens.

I have a nice thought experiment I like to do with people when confronted with "AI can't do x". Let's go back in time. How much do we need to go for this to become true? So let's try the 2000s.

Say you get a "fable/mythos/sol/gemini/kimi/glm/deepseek/whatever" in a box (and let's assume no guardrails). And you go back to the year 2005. It's "20 years ago", the world is slowly building back from the dotcom bubble, the Internet is really starting to happen, more and more things are interconnected, more and more things are connected to the Internet. Cool.

(for a bit of context, around that time we also saw the first high-impact worms like blaster that hit massive amounts of computers even reaching nuclear powerplants, we had a ton of ssh exploits that even made the movies, and security in general was a "nice to have")

I'd say that with the uber-model-in-a-box and a few prompts, you could reasonably make a case that you could design a worm that could infect 90-100% of the things connected to the Internet back then, stay as hidden as possible (in-memory stuff, vm execution, etc), move laterally into any network at inhuman speeds, and infiltrate almost every interconnected computer that has a link to the "public Internet".

Would that qualify as "take over the world"?

Then you could ask "what happens in 20 years from now?". And, thankfully, now we'll also have the AI on the blue side.