alt Hacker NewsFrom your link:
> for every doubling of model size the number of training tokens should also be doubled
Well, I don't think many claim that increasing the model size while also increasing the training tokens doesn't make sense.
The problem is how to get the training tokens...
Interestingly enough labs do train with 100s or even 1000s times more tokens than what's proposed by the paper as an optimal value.
And the answer to your question is basically just “synthetic data”.