Trying to solve (much less explore) mathematics using probabilistic next-token prediction seems like the really long way around, especially when we have pretty good deterministic tools available for our use. I don’t know why anyone would bother doing anything besides working on the correct manipulation of tools.

Brains have various structures that have distinct architectures. I don’t see any indication that the best way forward is to try to shoehorn everything into a single computational paradigm.

It’s like trying to make a flying submarine car. It might technically be possible, but it might not be worth the trouble, and it’s unlikely to result in a vehicle that works excellently in any of its environments.