alt Hacker NewsI don’t think this works, for a few reasons. First, intelligence gains from scaling the models bigger is sublinear now. So they could eke out a little extra performance, but the increased cost will eventually eclipse the economic value gained from this.
Second, humongous models are impractical even for them to deploy widely. They’re best used as teachers for smaller, more efficient models that can crank out the volume they need to sell.
Finally, there is a data wall. Sure, they can keep scaling RL on math problems and code. But with everything else, where will the supervision come from when they need several orders of magnitude more?
Replies
assuming the technology of model architectures does not gain any further breakthroughs that returns us back to the gains previously seen. I'm of the opinion that we still have some discoveries on the mathematical side of the fence to go that will improve models further.
I agree. And even if they were able to do it for one more round, it's not a sustainable strategy. What they (Anthropic and OpenAI) need to do is build platforms and integrate verticals.