Speculations Concerning the First Ultraintelligent Machine (1965) [pdf]

I think it's worth noting the date of this when people claim that AI is being rushed without asking any of the important questions of what it might mean for us. This has been a topic of serious discussion for many many years.

It feels like the people who laughed at these fanciful ideas then are the ones now pretending those same ideas are new revelations that have gone unconsidered.

> Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever.

That's a pretty early definition of what we now call ASI (Artificial Super Intelligence). In the next paragraph the author goes to describe what we today call the "singularity" (ASI designing better ASI). But that term seems to be associated to some very weird communities, so the concept is relegated to sci-fi. Even though we're already seeing signs of things we have working towards this. Interesting to see that in the past "Man" was more optimistic :)

> It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make, since it will lead to an “intelligence explosion.”

Well, that didn't happen.

> The first ultraintelligent machine will need to be ultraparallel, and is likely to be achieved with the help of a very large artificial neural net.

Right on, that we have.

> The required high degree of connectivity might be attained with the help of microminiature radio transmitters and receivers.

Hahaha, this is straight out of 60s-70s sci-fi, where their best futuristic interfaces were smaller CRT screens / flashy keys, etc.

> The first ultraintelligent machine will be educated partly by means of positive and negative reinforcement. The task of education will be eased if the machine is somewhat of a robot, sinae the activity of a robot is concrete. [...] the machine will be able to lem from experience, by means of positive and negative reinforcement, and the instruction of the machine will resemble that of a child.

Heh, nice early insights. They missed the how, but RL is the thing that ultimately made it "click" and be useful. And there is increasing talk about embodiment and how that'll help the next iteration of models. So there's that.

Overall a cool read. I skipped most of the middle part, only skimming for things here and there.

Up to five years ago[^1], we lived an age of pure speculation about intelligence and minds. That age brought about multiple theories of the mind, and to me, their finest legacy are arguments of the form "$X is not $concept because $theory", where any given day $X can be "LLM" or "LLM+harness+tool", "agent" and $concept can be "intelligent", "aware", "rational", "long-termed", , etc. Those arguments, in the context of $theory, are not wrong, and they are in fact quite useful. But there's only so much utility in a sieve.

We are going to need positive theories of the mind, and we will have to build them using terms and bits that avoid confrontation with the old guard. So, maybe instead of "mind" we will say "artilect"[^2][^3], and then move on to engineer desirable properties. For some of those properties ("alignment", "paucity", Kantian analysis and synthesis) we are already there or aiming for, but as we design to take on more and more challenging deeds, we may need to go back, steal and rebrand other old ideas. For example, "long term goals" and "self-preservation" are strongly linked in humans. Can we reify that relationship in the new artilects we make?

Other things we will need to make from whole cloth. For example, minds with an intuitive understanding of, and capability to manipulate: quantum mechanics, biological organisms, the workings of an ecosystem, the stock market, a stock company, or itself and other minds.

[^1]: YMMV, depending on where you draw the line between fine intelligence and stochastic parrots.

[^2]: Not the dictionary meaning, but the term as used by Alastair Reynolds in "Blue Remembered Earth (2012)"

[^3]: We may not use the term "LLM", because of the stochastic parrot crowd, because LLMs become one of many ingredients, or simply (but unlikely) because we find a completely different tool for the kit and discard LLMs entirely.

[dead]