alt Hacker News> LLMs are already more persuasive than most human writers.
And writers are well known to be the true holders of power in the world... In any event, you can see that even among humans, the most persuasive aren't always the most intelligent.
> the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
There are many addictive things, most of which have little to do with intelligence.
> because we can just build for hurricanes as if they're Poisson distributed
Hurricanes are not particularly important components of power in society. The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
> Humanity? All of it? Name one.
Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
> And writers are well known to be the true holders of power in the world...
Martin Luther, Adam Smith, Karl Marx. All kinda famous for things they wrote, which changed the world because of those writings.
> In any event, you can see that even among humans, the most persuasive aren't the most intelligent.
Can you? Again, correlation. The village idiots don't generally get heard; the professors tend to be.
> There are many addictive things, most of which have little to do with intelligence.
Missing my point: these are not natural things found in the wild whose addictive nature is pure coincidence of biology, these things were made addictive by applied intelligence.
> The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
An AI that has taken over doesn't need consensus, it only needs "the economy" to the extent that the economy provides specifically what it needs.
> Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
Yes. Look at all the people who keep listening to Musk despite https://en.wikipedia.org/wiki/List_of_predictions_for_autono...
Or same for Trump or Bush, both re-elected. Trump's second egregiously so, if you think people punish mistakes. And those two were both in a democracy; the British before universal suffrage had "Their's not to reason why, Their's not to make reply, Their's but to do and die"(https://en.wikisource.org/wiki/Maud,_and_other_poems/The_Cha...), while Chairman Mao had the disastrous Four Pests, Stalin the Holodomor, and Pol Pot… where do you even begin with him?
> Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
You said "biological species that are more capable of destroying humanity than the other way around".
None of those are more capable of destroying humanity than the other way around. We have, in fact, entirely eliminated e.g. smallpox, but no disease, not smallpox, not HIV, not even the Black Death, was ever capable of wiping out the entire human species. We've got plans for gene-drives to wipe out (malarial) mosquitoes and the only reason we're holding back is uncertainty about the side effects on the broader ecosystem. If anything, our biggest problem in this regard is wiping out species and then regretting it later.