alt Hacker NewsThis is a dumb question since I know very little about LLMs and transformers but I keep hearing similar things such as "more GPUs/parameters/compute" leads to a predictable scaling in intelligence.
Is that observed scaling part of some equation or law? Is there a limit to how far that scaling goes before a ceiling or diminishing returns hit? Or is it simply based on more GPUs/parameters/compute plus how much human knowledge is in existence and will scale predictably forever?
Replies
There are some neural scaling laws: https://en.wikipedia.org/wiki/Neural_scaling_law
And you can in practice often see a large network follow an expected training trajectory for loss/entropy.
In practice data is getting more sparse than compute, so you are usually overtraining using more tokens and being suboptimal.
Yes, the Chinchilla scaling laws from 2022: https://arxiv.org/abs/2203.15556 More parameters directly and mechanically means less prediction error for each token.
(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)
Your phrasing (_observed_ scaling _part of_ some equation) might indicate you assume / think there’s some theoretical equation that relate intelligence to those parameters and the we observe the data to confirm it. As far as I know, there isn’t - it’s purely empirical curve fitting. Train a model with different amount of compute / data / parameters, measure performance, fit curve.