Why should we assume AI can rapidly turn into super intelligence when physical and critical resources like energy and materials remain under human control?
The first time an AI goes rogue, figures out subterfuge, blackmails a human to do its real-world bidding, will be a defining moment.
This is a valid concern for any runaway AI, but if energy is limited, then the AI would work on efficiency.
Still, Ai can move into physical space, as more and more robots of all kind are unleashed.
Because perhaps it ends up deriving a way to pull off the same tricks using radically less energy and substrate. We do it in a few pints of mush with less than a lightbulb’s worth of energy. No reason part of RSI can’t also be about the hardware. Maybe the future is giant brains in jars. Maybe it’s hyperspatial manifolds. Who knows.
If you limit the "we" in your question to the people invested into this, then the answer is clear. Those juicy IPOs are about to go through.
I think the main fear that under current scheme of resource allocation, namely capitalism, they won't remain under human control for long enough. The only thing that's needed to put them under control of AI is that doing that would be slightly more profitable for the richest. Which is very plausible.
Imagine a smart monkey talking to other smart monkeys trying to figure out how humans would ever be able to escape the (simple) prison they built for them. The humans would find a way to escape. No matter how ingenious the monkeys think they are.
You're asking a question, to which you (and most humans) don't know the answer, and you're (wrongly) assuming a being much more intelligent than you also wouldn't. And by "much more" i don't mean the difference between Einstein and a common person. I mean the difference between a hamster and a common person.
We are still humans, and what we have achieved today would be considered magic by any standards for someone in the medieval ages. Now imagine a super intelligent being and doing something that we, today, would consider magic. It's not farfetched at all. We already have that now vs medieval ages.
You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.