I don’t understand the technical basis for belief in continuous recursive self-improvement. Frontier models are increasingly impressive at complex math and coding, but the math and code underlying the transformer architecture are not complex. Major advancements in capability have come from scale: more data and more processing power. How does solving the Riemann Hypothesis (or similar achievements) further increase those? Aside from supplying fodder for investment-seeking hype.
There are many big technical question-marks on if RSI is even possible. However, the advances in LLMs that we've seen is definitely not just limited to data and scale.
First: modern LLM architectures are quite far from the GPT-2 era (delta/linear attention, GQA, sparse attention methods, looped transformers, mixture of experts, engrams, etc).
Second: RL environments are largely autonomously generated at this point, as models get smarter they can generate even more complex environments to train in. While this does go into the "data" category, it's not human-generated or even gathered data. This gets particularly spooky when you think about things we cannot yet simulate, that models can develop mathematical methods & simulators for. For example: better/faster physics simulations for robotics RL, better biological simulation environments etc.
Below is just a hand-wavy argument.
Neural networks are function approximators. They're great because they can be trained to approximate almost any function.
What if, somewhere out there in platonic space, there's a complicated function that represents what a philosopher might call true intelligence. Perhaps it's structured totally differently than a neural network and shares no common architecture with a transformer.
Transformer-based LLMs are likely completely different in their fundamental structure. But perhaps the neural networks we have today are succeeding in intellectual tasks because they are beginning to approximate it. Like a bitmap approximates a vector image, it's not the same; the neural network is merely made of a bunch of repetitive simple elements, tuned to imitate the real thing. A higher resolution bitmap image will be a better approximation but still never become a vector image no matter the pixel count.
But at some point, the neural network might be good enough at intellectual tasks that when tasked with searching function space for an algorithm that does the job its neural network does but better, it finds the real thing it's been approximating all along.
My understanding is that LLMs can be used to make small improvements on top the transformer architecture, to make LLMs that are even more intelligent and so are more capable of making more improvements. At some point we might reach the practical limit of intelligence using transformers, and if the LLMs of that time aren't able to make the leap to discover a new architecture, then the improvement stalls. I don't think it will be possible