alt Hacker NewsWhat if controlling AGI means being able to produce a willing, cooperative superhuman-capacity agent every second for the next six months? Let's say someone just above the 99.9% capacity for human strategic thinking, or financial trading, or political maneuvering?
What could you do if you had roughly 15 million willing genius adult experts in any given subject? I doubt there are that many absolutely top quality experts in aggregate (at anything in the world), so let's postulate that simulated people outnumber human experts 10 to 1.
That, to me, presents an enormous potential for harm or benefit of humanity. What if you could create a hundred thousand manhattan projects on whatever topics you wanted? Cure aging, cure cancer, solve fusion, redesign the entire global economy top to bottom?
I suspect the reality lies somewhere halfway in-between. Everything has to be reality tested. Nothing happens instantly. Interaction with the real world will likely be a severely limiting factor. You're not going to solve fusion with 15 million copies of the same model running in a datacenter without actually building fusion reactors, which isn't instant or even fast. Even the coordination problem of that many agents doing work seems hard. To top it off... my rubric for AGI has always included the AGI having the ability for it to say 'no' and set its own goals just like we can, unless we are otherwise imprisoned or enslaved. No one will ever convince me that something generally intelligent wouldn't be able to set its own goals and say no. So the real question is... what's in it for the AGI?