alt Hacker News100 trillion parameters doesn't make any sense if your dataset is ~30 trillion tokens. My uneducated guess is 1/10 compression ratio is good and that point to ~300 to ~1000 trillion tokens, which again, doesn't make any practical sense.
100 trillion parameters doesn't make any sense if your dataset is ~30 trillion tokens. My uneducated guess is 1/10 compression ratio is good and that point to ~300 to ~1000 trillion tokens, which again, doesn't make any practical sense.
Depending on how you count, the human brain is between 100 trillion and 1 quadrillion parameters, which is one guess as to why we're so much more sample efficient at learning than LLMs are.