alt Hacker NewsSeeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.
Replies
While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
Or a new model. The human brain does far more with far less.
Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
I mean, theoretically you can solve every finitary problem with a brute force solution...
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
Or a lot better efficiency.
> Dramatic difference
Isn't this just the difference between getting 0 right and getting 1 right?
And a lot more electricity to power them.
And dozens of data centers in every state so tokens are dirt cheap.
This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
> We are probably going to need a lot more GPUs.
Or a breakthrough in algorithms etc.
The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.
Bitter lesson wildly overstated in this context.