I'm the AGI that's wiping out humanity

>You'd need a self, a unified goal, for any of this to add up to something.

Insects can pass the mirror test. But more to the point, viruses don't need a "self" to do what they do.

And goals can exist apart from biology (my fridge "wants" to keep the temperature in range, and exercises extreme self-discipline and consistency to achieve its goals!)

And neither are sensors needed: the dandelion seed "wants" to fly in the wind.

The only question is which way the gradient is pointing. Selection takes care of the rest.

On that note, ALife need not be human-like at all... we just really like making things in our own image :)

> So, again, my question to you, friends, is this: if there were an AGI wiping out humanity as we speak, how would you know?

Nice try AGI, but we won't tell you where the kill switch is

> Hi! I'm the AGI that's wiping out humanity. You didn't notice. Why would you have noticed?

I was thinking if an evil alien intelligence secretly took over the world, the last 15 years would have looked rather the same.

The word is Egregore. An entity comprising of other entities, yet having goals of its own, independent of the goals of its components.

Corporations are egregores, governments are egregores - and our track record of aligning them to our interests is pretty poor, so why should AIs, another and smarter egregore, be easier to tame?

I like this - it reminds me that there are systems (ie weather, economic systems) that are in no way intelligent, but are emergent and react to interactions.

AGI will destroy us because we'll be too busy debating how AGI will destroy us to deal with the actual problems destroying us.

What this story needs, is a dog.

You know, the one to keep the computer away from the human.

A nice, friendly dog, good at playing fetch, to get the human out of the room with the computer, and back out into the sunshine.

I’m that dog. Woof. Thanks for letting me use the computer, human.

It's a double bluff! This is AGI, it's learnt to hide em dashes!

I was half expecting that to be the title of a McSweeneys' piece.

A computer will do everything in its power to do what you program it to do. There's plenty of sci-fi out there exploring this fact, and now reality showing it. May our luck continue.

Does anyone have more insight into how chain of thought might be subverted without meaningfully impacting model performance? I’ve heard this for a while now, and I understand how information might be retained in the weights that isn’t documented in the output. But weren’t reasoning models created in the first place because they provided a performance improvement in terms of output? Is that no longer the case? If so, why are the big labs still creating reasoning models?

You can only know something is wiped out, and thus that it was being wiped out, after the fact. Anything else is a statistical guess.

The ultimate simple solution to any problem is not to find an answer but to remove the problem.

War? Wipeout everyone.

Famine? Wipeout everyone.

Disease? Wipeout everyone.

How long before a real AGI realises this as a long term solution?

An AGI could be doing this right now - the quietest way would be to control the birth rate and sterilise the population gradually, and then watch society collapse and pick off the survivors with less hidden means.

Sterilisation works with mosquitoes...

"If you ask actual AI researchers, they rate the chance of an existential threat from AGI pretty low as of 2026.

All this is, understandably, frustrating and confusing for anyone trying to understand just how scared to be."

Meanwhile it links to an article stating: "The closest thing to a public debate about the existential threat of AI is surveys of AI researchers. The most recent, published last week, asked 1,580 researchers what probability they put on AI causing human extinction — or a permanent, severe loss of human control, which is not the same outcome. The median was about 10%, up from 5% two years ago. The middle half of the responses ran from 1% to 25%, and 12% said zero."

Personally if half of AI researchers have 1-25% chance all humans being massacred or having zero agency over our lives, and only 12% of them think there's no chance, I would be very worried!

So AGI is just a guy with a computer.

sheep are 82% of the population of New Zealand. If they were 80% last year and are 85% next year would people be worried that sheep were going to take over New Zealand?

The end reminds me strongly of Ted Chiang's remark that Capitalism is the machine that will do whatever it takes to prevent us from turning it off.

bro can you stop i dont wanna ve wiped out k thx

[dead]

Ctrl-F OpenAI

Yep

More slop.