Synthetic data is key here! Compared to 5 years ago, we now have oracles that can generate massive data sets of perfectly labeled multimodal data, practically for free. The number of architectures that can benefit from that is innumerable, and far beyond just LLMs themselves. On top of this, LLMs can implement any architectural ideas you have, and write custom tools to manage training and evaluation.

Whether or not LLMs can self-improve their frontier capabilities, they can absolutely create a wake for themselves that accelerates everything else that's training on their synthetic data. We'll see every architecture of the past 40 years suddenly show leaps and bounds.