My take- our digital knowledge and simulation systems are constrained by our species existing knowledge systems- language and math- despite our species likely living in a more complicated and indefinable universe than just language and math.

Ergo the simulations we construct will always be at a lower lever of reality unless we "crack" the universe and likely always at a lower level of understanding than us. Until we develop holistic knowledge systems that compete with and represent our level of understanding and existence simulation will always be analogous to understanding but not identical.

Ergo they will probably not reach a stage where they will be trusted with or capable enough to make major societal decisions without massive breakthroughs in our understanding of the universe that can be translated to simulation (If we are ever able to achieve these things)(I don't want to peel back the curtain that far- I just want a return to video stores and friday night pizza).

We are likely in for serious regulation after the first moderate ai management catastrophe. We won't suddenly go from nothing to entrusting the currently nonexistent global police (UN)(lol) to give AI access to all the nukes and the resources to turn us all into grey goo overnight. Also as initially AI control will be more regional countries will see it as a strategic advantage to avoid catastrophic AI failures (eg AI chernobyl) seen in other competing states- therefore culture of regulation as a global trend for independent states seems inevitable.

Even if you think there is one rogue breakaway state with no regulation and supercedent intelligence you don't think it takes time to industrialise accordingly and the global community would react incredibly strongly- and they would only have the labour and resources of their states to enact their confusingly suicidal urges? No intelligence can get around logistic and labour and resources and time. There's no algorithm that moves and refines steel to create killer robots at 1000 death bots a second from nothing within 2 weeks that is immune to global community action.

As for AI fuelled terrifying insights into our existence- we will likely have enough time to react and rationalise and contextualise them before they pervert our reality. No one really had an issue with us being a bunch of atoms anyway- they just kept finding meaning and going to concerts and being sleazy.

(FP Analytics has a great hypothetical where a hydropower dam in Brazil going bust from AI in the early 2030s is a catalyst for very strict global ai policy) https://fpanalytics.foreignpolicy.com/2025/03/03/artificial-...

From my threaded comment:

============================= LLMs are also anthrocentric simulation- like computers- and are likely not a step towards holistic universally aligned intelligence.

Different alien species would have simulations built on their computational, senses, and communication systems which are also not aligned with holistic simulation at all- despite both ours and the hypothetical species being made as products of the holistic universe.

Ergo maybe we are unlikely to crack true agi unless we crack the universe. -> True simulation is creation? =============================

The whole point of democracy and all the wars we fought to get here and all the wars we might fight to keep it that way is that power rests with the people. It's democracy not technocracy.

Take a deep breath and re-centre yourself. This world is weird and terrifying but it isn't impossible to understand.