GitHub dumps are about 115 terabytes. The common crawl is in the petabyte range uncompressed for every year. Apparently there are dumps of Reddit too in spite of their efforts to ban bots and it's not solely due to the use of residential proxies. For a 1:20 parameter to token ratio, you can still train up to 10 trillion parameters so 10T parameters times 20 is about 200 trillion tokens. Then each token is 4 bytes so 200 times 4 is about 800 terabytes, which is not inconceivable, the common crawl alone has more data than that. So does the internet archive if you donate to them, Anna's archive is 2 petabytes including images, etc etc not all of it is text, but training on multimodal data increases model intelligence by virtue of being multimodal