It does feel very brute force though, doesn't it? It feels like we should be able to guess the manifold that has all the important stuff for each question. But it doesn't seem like we can currently! Why not?

It is kind of terrible either way, I feel.

We're making enormous investments, that require enormous profits to justify this. And it's not just big bad rich people doing this, I shudder to think what the exposure of my pension portfolio is to this stuff.

So either there will be no enormous returns, big flop, big recession. Probably terrible for tech workers, because it is our employers who are most exposed.

Or it will make enormous profits. But how? By replacing humans. If it is really worth all this money, then surely the owners of this stuff will suddenly own all the intellectual productivity in the world. And I have a feeling they won't really share it with the "share" holders.

Compute is a convergent instrumental goal. Any runaway paperclip maximizing AI is going to want lots of GPUs. And any runaway corporation trying to build an "aligned" runaway AI is going to try and feed it as many of them as possible.