alt Hacker News> Current top models are mere (mere in big datacenter terms) trillions of parameters right now. ... We'll soon start hitting 10's of trillions and 100's of trillions of parameter models.
Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.
We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).
FTA "The core philosophy there is scaling, and aggressively not research."
This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).
I'm not an AI booster (in a "should" not "can" sense), but in terms of "can" we are going to see next generation hardware (i.e. Vera Rubin GPU architecture) and improvements in model density (i.e. loosely improvements in intelligence per millions of weights), so the models will get bigger, the models will get more dense, and we may see more "expert models" i.e. in the sense that for a while there were human generalists that knew all of science and mathematics until the 1800s, after a certain point we had to specialize since there was too much to know. The models will need to partially specialize after hitting the hardware ceiling.
Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.
I don't think we should do this, but I feel doomed in that I suspect that we "can".