alt Hacker NewsI think the moat is going to be compute. So far compute needed to push the frontier is still extremely cheap so the capital can afford to spread its bets. But when further improvement is going to cost in trillions, capital will have to pick a winner and bet only on him. It won't be a matter of finding the best bet, it will be a matter of survival.
This will cause the picked winner to get massively ahead with sheer compute alone used both for training and inference dedicated to recursive self improvement.
Replies
Surely there is a point where algorithmic improvements will be more cost effective than buying more hardware.
But the old models still exist at trivial marginal cost. The frontier models would need to dominate every price point to really take all and so far they haven't been.
Not just compute but energy. Most of Europe has no access to the cost effective power generation needed
I guess all predictions age like milk, but here's one:
There's a law of diminishing returns at play here, and doubling the energy cost of training to wring 2% more performance out of the technology isn't going to be very useful, because most of the problems it is capable of solving will be solvable with the previous-gen 98%-as-good model.
("there's a law of diminishing returns at play here" is an article of faith. But then, so is the belief that these models will keep getting better).