Pre-training already makes up a very small cost of the training now. Post-training makes up ~90% of the training cost as modern LLMs are trained using RLVR on synthetic data. People still think that a lot of it is coming from scraping wikipedia like it is 2023.
alt Hacker News
Pre-training already makes up a very small cost of the training now. Post-training makes up ~90% of the training cost as modern LLMs are trained using RLVR on synthetic data. People still think that a lot of it is coming from scraping wikipedia like it is 2023.