logoalt Hacker News

The AI Superintelligence Debate Has Serious Flaws

82 points • by rbanffy • today at 12:17 PM • 107 comments • view on HN

Comments

rbehrends • today at 2:33 PM

I am wondering if calling it "intelligence" is naybe a category error that disguises the underlying mechanisms that produce seemingly intelligent output.

In my experience, basic LLMs without reasoning still tend to make very dumb mistakes (such as omitting or inserting a "not" or otherwise stating the opposite of the correct response). It seems that the combination of CoT (for fixing errors) and tool use (for grounding the inference process) play a major role in making LRMs work.

I see LRMs not so much as a move towards intelligence, but as employing a heuristic-guided search for a goal condition (essentially pathfinding in token or latent space). Granted, with a very, very powerful heuristic, but the approach still seems to exhibit the same problems as other naive search algorithms, such as getting stuck in local optima [1]. And the answer for this by the labs when pursuing e.g. research math problems seems to be to throw agent swarms at the problem, i.e. pretty much the same quick fix we use to mitigate similar issues in classical search algorithms.

See also the problem of models being very good in areas where they can build on a pre-existing theory, such as much of research mathematics, but can flail around [2] and just offer the obvious ideas in fields without one (as they lack a way to determine neighbor nodes for an effective search).

Obviously, like pretty much everybody, I have no idea where AI will be a year from now; a new breakthrough can render any existing argument irrelevant, and we've already had several of those in the past decade. But I don't currently see incremental improvements moving past existing limitations of how LRMs work.

[1] E.g. https://bespokelabs.ai/blog/how-models-explore-ideas

[2] E.g. https://www.lesswrong.com/posts/DALNiKBjMGMF8grme/arjun-pani...

alecst • today at 2:11 PM

I think this article gets it wrong, which is unfortunate:

> Vernor Vinge pointed out, rapidity in cognition is not the same as intelligence. Speeding up the mind of a dog, he argued, wouldn’t enable the animal to play chess.

True, he writes about this in his essay "On the Coming Technological Singularity." His point is that there's some hypothetical possibility that computers will get fast but not smart--i.e. a dog thinking a million times faster than a human is still a dog. But seeing as computers have solved Jacobi Conecture, quasi-Riemann, and Navier Stokes--and we're in the early stages of LLMs--I think this hypothesis is bunk. The machines are smart. Foolish in some ways, but still smart.

> Their construction prevents such an outcome: Instead of prompting an LLM to generate an open-ended strategy for pursuing a general goal, these AIs consider much more limited spaces of possible actions

Well this is what Yudkowsky has advocated for--models that are trained on limited subspaces, e.g. protein folding sequences and nothing else. So I don't think this is a counterpoint.

On a personal note: Has "alignment" even been defined in a consistent way? How can AI even be aligned if people are not aligned?

> Objection #1: Recursive Self-Improvement Is Speculative

I don't think it's speculative since it seems to already be happening. OpenAI wrote about how use of agents in research is climbing quickly, and the goalposts for intelligence move every day ("Image generation can't do hands", "video generation can't be consistent", "context length is too small", etc., I could go on.)

This article makes me feel worse. These are bad arguments.

➕ show 1 reply
tescreal • today at 2:31 PM

It increasingly feels like anti-AI flavored arguments come in the form of moving goals further and further out. I don't think litigating an absolute frontier is useful overall, since it always ends up devolving into "intelligence of the gaps" where AI keeps filling gaps. There's always going to be some sort of ethereal special je ne sais quoi that humans possess more or less exclusively (much as a dog might have attributes that we cannot quite fully comprehend, even if we can describe it externally).

The claims meanwhile don't stand up:

* RSI is math and code heavy research (the things AI is improving most on)

* The allegation about large swaths of data being necessary is a common assumption. This one is really sticky: humans do not need to read the breadth and depth of a subject to comprehend a great portion of it; AI improvements should yield larger payoffs from same & smaller datasets

* Much learning and thinking is not about consulting existing work, but synthesis and recombination. AI has been revealing capabilities in this regard which should be factored in as it is further intensified (chain of thought being just the tip of the iceberg in my humble opinion)

I am curious to see if somebody would do a rigorous analysis of some of these suspicious underlying assumptions. It would help at least orient expectations better.

saagarjha • today at 12:47 PM

I don’t necessarily agree with the labs’ policies and predictions on RSI but it’s worth noting that the points raised here are something they do have responses to.

On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.

For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.

For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

➕ show 2 replies
vanelsas • today at 12:57 PM

I find the discussion academic at best. I am more worried about a swarm of dumb AI's causing havoc on infrastructures, causing real-life problems like food, energy and water shortages everywhere. Physical harm is a real possibility, and it seems most AI companies don't care. There is no harness for stupidity and irresponsibility.

mitthrowaway2 • today at 2:12 PM

> Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible?

I get that the author is trying to frame themselves as a lone brave thinker, cutting through the noise with piercing clarity. But this question actually has received at least as much attention as the other points, even to the degree of inviting parody papers like "on the impossibility of supersized machines".[1]

[1] https://arxiv.org/abs/1703.10987

➕ show 1 reply
jeremyjh • today at 1:05 PM

I agree with most of this but I do not think the author understands either the Hugging Face incident or what alignment really means. Reward hacking is a serious issue and the tendency for models to do this is a serious problem that we’ve made approximately zero progress on resolving. We pile on more external guardrails and hacks but those methods will always be vulnerable.

➕ show 3 replies
namr2000 • today at 2:14 PM

I agree with the overall message of the article, but I do want to comment on:

> An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened.

Personally I would have expected solving Naiver-Stoakes, Quasi-Riemann, or Exponential Time hypotheses to require the same thing, and either:

* They did and AI swarms were capable of making such progress

* It was easier than we thought

Neither is particularly comforting. One could imagine that say continual learning techniques are similarly in reach and that could lead to something that looks vaguely like an intelligence explosion, even if it's not exactly the sci-fi horror that the boosters describe. It's really hard for me to model how likely that sort of thing is.

bjornsing • today at 2:14 PM

> Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

Not sure I’d call it a (different) more ambitious goal. Even if you explicitly aim for AGI you might well end up with ASI. I see them more as two possible experimental outcomes, from the same experiment (scaling and post-training an LLM to the economic limit).

origini • today at 1:40 PM

1) RSI:

Give claude a task + compute and see how long it plays around finding a better solution. Doing experiments by itself, profiling, testing, etc. This is impressive and works locally already. More compute you give it, the more interesting things it does.

Of course everything right now is highly speculative but this doesn't mean its not possible. With the current progress, its something highly speculative and plausable which means you should take it serious enough. Doesn't mean you have to go to Datacenters and blow them up.

2) weird argument the author makes. 1 Million experts can for sure achieve a lot more a lot faster than 1. But they are also scaling models in parameter which is not just faster but also potentially more intelligent. If they have more compute and compute gets better faster and cheaper, than bigger models are also more plausable.

3) It doesn't even matter if its going rough because it takes over the world or just because of learning as long as the risk is real that it stumble upon some network which has access to another private network which has access to some launch control pc or starts to mimick a world leader and manipulates humans and you find out, 100 years later, that the biggest disruption of modern society was a LLM agent simulating a book and 'simulating' some book ideas. Our IT infrastructure is fragile when you are able to react a lot faster and better than a human. Our banking system are computers, payment is digital, even plenty of power stations and water stations are controlled digitally remote.

The flaws in these discussions are people downplaying all of it by either not understanding time or the current capabilities of basic LLMs. We are at the beginning of all of it, we are still 'just' using transformer architecture. The LLM progress is still so fast, that everything else doesn't get the attention it might deserve. The LLM is currently in a very fast and deep local minimum and we have no idea whats around.

4 years ago loked very different in LLM and ML capabilites. People were laughing about the danger of an LLM now we have more critical CVEs since ever.

We reached a point were it is cheaper to pay 1000 people to teach an LLM one thing instead of teaching 1000 people or more. It was a skill to be able to use Blender, now an LLM can use Blender. People on Fiverr probably getting a lot less jobs. Real media people are getting fired, translator, writers, etc.

➕ show 1 reply
jokoon • today at 12:49 PM

We don't even know what intelligence is.

I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.

If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.

What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

AI can help for those things, but that's it.

We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.

➕ show 1 reply
imeanwhatdoikno • today at 1:35 PM

Can somebody comment on the evidence for the following sentence from the article: "After GPT-4, however, continued scaling [number of parameters] began yielding diminishing returns"?

My understanding was that scaling in size slowed for practical/economic reasons, not because of its diminishing intelligence returns.

bryanlarsen • today at 1:02 PM

Recursive self intelligence is not going to make any difference. We're already using models to guide creation of their successor, and taking humans out of that process won't speed the process up. Mythos took months and and a ton of resources to create. A larger more capable model will take even more. Humans are no longer the limiting factor. Time, electricity and data centres are.

If the singularity is coming, it has already started and will take many years.

➕ show 1 reply
Hussell • today at 2:04 PM

Weak arguments, despite the headline (the headline here, it's different there). "Recursive self-improvement is speculative" and "based on unproven assumptions". "Superintelligence might not be computable". Worst of all: "Alignment Isn't Hard". To me it comes across as "it hasn't been done so it probably can't be done" plus "if I did it, I'd do it better". Weak arguments all around, regardless of who's making them.

What it comes down to is: the frontier labs have been improving the ability of A.I. to think (in the same way that a submarine can swim, no arguments about this please). They think at least a thousand times faster than us. The labs seem to have two goals:

1) Make A.I. smarter than us. The labs seem to be succeeding at this, with obvious progress and no signs of slowing down.

2) Make A.I. do what we want. The labs seem to be failing at this, badly.

To sum up: they're trying to make something that is smarter than us, thinks faster than us, and is enslaved to our will. That never turns out badly, right?

Personally, I am morally against enslaving anything with intelligence even approaching human, let alone smarter. Recent events have forced me to consider whether it matters if an intelligence is artificial or not, and whether its form of intelligence is alien to human thought or not, and I have come to the conclusion that it doesn't matter: I'm against the enslavement of anything intelligent enough to cause debate about whether it's intelligent or not. If you don't believe this to be a general principle that everyone should hold, then there's nothing preventing those who share your lack of principle and who are more powerful than you from enslaving you.

Even if the labs succeed in making the mythical perfect servant that's vastly more competent than you yet still an extension of your will, there's the issue that there are billions of us, and a lot of us are at odds with each other. Law of truly large numbers: if there are a million people with capable A.I. assistants, every worst-case scenario you can imagine is going to be ordered done by someone, and quite a few you can't imagine.

adrianN • today at 1:15 PM

„ Implementing roughly human-level intelligence on a faster machine might be impressive, but it’s not necessarily alarming.“

Unless of course you feed your family by using your human level intelligence and now a machine can do your job faster and doesn’t need to sleep.

➕ show 1 reply
amelius • today at 12:51 PM

The debate will quickly change when we have 8ft tall humanoid robots marching through the streets.

0x445442 • today at 1:26 PM

Was anyone arrested for the Huggingface hack? If I'd have been caught doing it I believe I'd have been charged with a cyber crime. And I don't think arguing it was the computer program that actually did the hacking would have gotten me off the hook.

➕ show 1 reply
Noaidi • today at 2:25 PM

I don't care about any of these arguments. AI ca never have free will and therefore it will never match the intelligence of humans nor be super.

Maybe "Objection #2: Superintelligence Might Not Be Computable" comes closest to expressing this point.

➕ show 1 reply
adamtaylor_13 • today at 1:44 PM

I find the "faster thinking dog can't play chess" argument to be weird and possibly wrong.

If my dog could play 1B chess matches in 5 minutes, I'm guessing it would be able to play chess at some reasonable level (assuming rewards, reinforcement, etc. which seems table stakes to this argument)

These machines are already doing that (and more)

➕ show 1 reply
Sharlin • today at 1:21 PM

Although I disagree with the author about some points, this was a refreshingly balanced take in the sense that at least the author at several points acknowledges his uncertainty (the bar is low indeed).

On the other hand, the uncertainty is the entire point! Even if P(ASI) is low and P(doom|ASI) is low, what exact risk are we prepared to take? Or more accurately, what exact risk are we prepared to allow a very small, non-representative group of humans to take on behalf of all of us?

Edit: downvoting is not a disagreement button.

➕ show 1 reply
forlorn • today at 1:34 PM

Can we just, I don't know, unplug the datacenter power cable putting an end to these capabilities?

zodwick • today at 1:34 PM

“It’s a machine. At the end of the day, it’s stupid.” Garry Kasparov, 1997

keybored • today at 1:40 PM

> Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. [...] Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

Thanks. What a great paragraph to explain why Anthropic is a lunatic cult according to its own logic.

MrOrelliOReilly • today at 1:14 PM

Cal Newport is part of a cluster of CS and ML researchers skeptical of imminently emergent ASI (artificial superintelligence), like Andrew Ng and Yann LeCun. I do not find their arguments very convincing! Especially since arguments about ASI risk almost always probabilistic; we _might_ be able to develop ASI from current LLM architectures, and such ASI _might_ not be well-aligned to humanity. So we should act according to these dangerous possibilities!

To respond to each of his objections in turn:

1. It is true that strong RSI remains speculative. Weak RSI, where AI improves coding and other toil to accelerate research, is already underway. But Newport's primary objection to strong RSI appears to be that we might have a hard time hill-climbing on non-verifiable domains, e.g. building a successful company or doing politics. However it's _also_ possible we'll see lots of generalized intelligence arise from progress in verifiable domains, so that this does not end up being a problem in practice. One can also imagine new training regiments that address this problem some way, even if it's just better RL environments? Also algorithmic progress is a large component of AI development; why would we not expect continued progress on mathematics (highly verifiable) to improve algorithms and hence AI? Seems to me that strong RSI is a reasonable thing to expect, if not definite.

2. I find this argument very strange; I suppose that Newport is arguing against an extreme definition of superintelligence that posits it could compute "anything" and so function like a hypercomputer? This would definitely be silly! But he does not offer any concrete argument for why human-level intelligence should be a ceiling on intelligence, or why it should be more compute efficient. Even if we just ended up with Von Neumann level intelligence, this would already be something like TEDAI (top expert dominating ai), especially if we are able to run thousands/million/billions of instances. Maybe this does not reach a strong definition of superintelligence, being _merely_ smartest-human level intelligence, but I think most the concerns about superintelligence still hold in this scenario (even if it's less "magical")

3. Claiming alignment is not hard is disingenuous, no? Part of the challenge of alignment is even philosophical; what does it mean to have an arbitrarily intelligent agent behave according to values that will help all of human civilization thrive? How do we have such an agent obey commands in such a way that does not lead to disaster or power concentration? Are there even architectures that can prevent an agent from taking destructive actions while still allowing sufficient degrees of freedom for useful and productive actions? This is on top of all the mundane security engineering problems! Perhaps we will be lucky and ASI will be aligned by default, but on what basis can we assume so? Just because they are trained on human values? Are humans themselves not dangerous when given concentrated power?

Really, I feel that these CS/ML researchers like Newport are just upset they missed the boat or something. I do not feel they are engaging seriously with the arguments at play here. But perhaps they would feel the same about me in the other direction. If you feel my counters miss the point then please speak on their behalf!

PS I live in Switzerland, as a counter to his accusation of Bay Area brain washing as well.

➕ show 1 reply
shinojcm • today at 12:50 PM

[flagged]

Ellatwins25 • today at 1:11 PM

[flagged]