alt Hacker NewsThere are some neural scaling laws: https://en.wikipedia.org/wiki/Neural_scaling_law
And you can in practice often see a large network follow an expected training trajectory for loss/entropy.
In practice data is getting more sparse than compute, so you are usually overtraining using more tokens and being suboptimal.
It should be noted that these laws aren't, well, laws; "a neural scaling law is an empirical scaling law that describes how neural network performance changes as key factors are scaled up or down.". Notice the word empirical here.
These empirical observations on quantities which aren't "intelligence" since no one has yet defined intelligence.