alt Hacker NewsI couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
Replies
If AI is now ascending an economic learning curve from:
1. Extremely useful (Claude Code & Waymo now)
2. Doing ~everything we do (AGI & Optimus in a few years? 10?)
3. RSI (?)
4. Being smarter than any living person at every intellectual task (?)
5. Being smarter than the best-organized aggregate of all humans (10-100 years?)
...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?
Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.
Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
> and saying that your technology has the power to cause a massive catastrophe
Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not in the same galaxy as "ready". What would that look like?
[0] https://en.wikipedia.org/wiki/Human
[1] https://en.wikipedia.org/wiki/Transistor
[2] https://en.wikipedia.org/wiki/Integrated_circuit
[3] https://en.wikipedia.org/wiki/Large_language_model