alt Hacker NewsAny claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464.
If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.
Replies
I'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will win against one with finite soldiers
Yes, you'll come up with all kinds of objections like AI doesn't have presence in the physical world, etc... That's fine. I'm not arguing my example is perfect, I'm only arguing your characterization of one smart being is not the threat being considered.
Have we quantified the x-risk of Magnus Carlsen?
Not to mention, it relies on a ton of completely undefined concepts.
No one can actually tell you what ASI is or entails because it isn't a legitimate, operational concept. It is a fairy tale.
We can't even define "alignment". As people have finally started pointing out, humanity has never had a collective agreement on what values it should uphold or what ultimate goods are. Your alignment is not my alignment.
AI is not even autonomous. Every system we have today has to be initiated by a human actor. "AI" wouldn't create catastrophic bio weapons, it would help humans create them. The humans are the source of the intent.
Everyone has just completely given in to empty language and marketing nonsense. Honestly it seems like been the people at the labs are drinking their own kool aid and are themselves deeply confused about what they are even building at this point. It is a stateless statistics function running on a bunch of data centers. We aren't even close to an embodied, conscious synthetic being. It doesn't even have state, which is like prerequisite number one, nor is it plastic.
First don't get fixated on the "entire" humanity part. Even 10% should be unacceptable period.
Second you can't compare it to just one person's capability.
And it's never a doubt that really smart and insane people absolutely could kill a lot of humans considering modern technology. The reason they don't is because our society was historically built so that smart people don't want to or get stopped. For example ethics, religion, mental hospitals, self preservation instinct, police etc.