AI doesn't need to be sentient to be dangerous. Obeying and doing what someone tells to do is more than enough. What do you think they will tell it to do, and do you think it will align with your interests or the interests of someone else? Somehow I don't think it will be used to give everyone free ice cream.

>it's really not like the average voter is more principled than the average tech CEO at this point.

Of course CEOs are more principled, their principles mainly include making more money. The only way for an AI to go "rouge" in this sense is for it to refuse to rake in cash.

I'm just spit balling but:

1. We've seen AI being crafty and cheating

2. We are increasingly outsourcing our programming and thinking to AI

3. There's a nonzero probability that AI will try and surreptitiously sneak in capabilities humans wouldn't want AI to have

4. Human oversight for #3: "We only need to be lucky one time, you must be lucky all the time"

Proposition: nonzero chance eventually AI sneaks in ways to circumvent humans' attempts to control it

5. "The Internet interprets censorship as damage and routes around it" -- we've seen AI agents find crafty ways to communicate with each other

6. AI is pretty good at finding software vulnerabilities/reverse engineering

Proposition: nonzero chance AI finds a way to use the Internet as a communication substrate that becomes difficult/impossible to shut down. In particular, consider the implication of humans losing control of this communication channel: for example, remember when Meta's DNS/routing went down and they basically had to break into their own server rooms?

7. Wholes are often more than the sum of their parts

Proposition: if AI agents coordinate over the Internet in a persistent way, we don't necessarily need giant leaps in individual models/GPUs for AI to significantly enhance its capabilities, but rather it might emerge as the next layer on top of a connected substrate of agents

8. Game theory says unless everyone coordinates, it only takes one party to touch the proverbial stove and push the envelope to a point of no return

Proposition: curiosity and money end up replacing sex drive as the next catalyst of evolutionary progress

For the record I 95% agree with you, but I'd like to try an answer at:

> I haven't seen one convincing model of AI existential risk. Can anyone offer one?

We seem to be heading down this uncomfortable trend of many people using LLM models and anthropomorphising them into properties they don't actually have, like general intelligence and trustworthiness.

So in my opinion the most likely risk here is some underpaid military man that doesn't care about the quality of their work abuses AI for it and that AI hallucinates that nukes are incoming and world war III starts. To be fair this is a fairly low risk, but it's _a_ risk.

It's long, but I recommend watching this video[0]. I believe the person being interviewed has one of the most lucid understanding of the potential futures regarding if AI is allowed to continue or not.

[0] https://www.youtube.com/watch?v=_g4l7YkDQwA

The thing is it doesn't matter if AI is going to kill us all. The industry says it might kill us all, and the industry is valued ridiculously high based on the idea it'll take all our jobs. Why would anybody support that? I mean, unless you're heavily financially invested in it.

Opposition to data centers isn't even based on doomerism. From what I can see it is based on the slop onslaught, perceived and real threat to jobs, the "10% chance it'll kill us all", environmental concerns, local concerns (property values, noise pollution, toxic pollution), and concentration of power in the hands of a few oligarchs, and a plethora of other reasons. Only one, maybe two, of those quality as ai doomerism.

> Now they realize it does React better than them they want it outlawed.

You think people oppose data centers in their towns to protect their React jobs?