> 2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".

Actually your comment made me sign up for an account just so I could say this is the real reason why AI won't take over in the way you say. This kind of stuff requires an enormous amount of experimentation. You can ask any theoretical physicist or chemist versus an experimental one and the conclusion is the experimental people actually find out what happens and how the great puzzle of the universe is solved. And humans could just refuse to collaborate. But that's the big weakness with AI I think it has no real world knowledge or empirical experience.