alt Hacker NewsI think the author did not mention that the way humans construct the knowledge ladder requires low complexity because each step gives power to the next, but the AI with the exponential exploration can give biggers steps but then it stagnates because the next step can be beyond the exponential exploration complexity. In chess a good strategy can be the best tool, in the game of go our experience suggest the same, but it could happen that mathematical thinking requires a type of policy that could be beyond the current ideas. I can not fathom a LLM could conceive from scratch concepts like the real numbers by Dedekind.
Just to add that the Dedekind cut example seems to stand beyond any RL policy used in chess or go, AlphaZero or AlphaProof. In the classical RL there is an state-action space. If the solution requires jumping to a totally difference action space (that must be created) the local policy stalls. Dedekind cut is an example of a out-of-distribution state-space generation.