alt Hacker NewsEven before AI we used to say if you write code that you only barely understand, then it will be to complicated to debug. (and/or maintain)
Mochizuki was still one human and it required legions of other humans to unpack and untangle to confirm that it didn't lead to anywhere in particular.
AI is now capable of constructions so complex that no human or human team can unpack. And its ability to increase that complexity is growing while our human ability is stagnant.
meta-AI analysis cannot help. We (software professionals who use AI regularly) already know that if you run into a situation where a Fable/Astra-generated analysis reaches the limits of our comprehension/complexity due to their subjectivity, throwing more AI at the problem doesn't always converge.
There are many reasons to feel optimistic about AI, and ultimately its general ability to help science and mathematics.
I see no reason to feel optimistic about the future of mathematics and AI based on the current path of frontier labs, unless the misalignment Tao is writing about can be reconciled.
Replies
> AI is now capable of constructions so complex that no human or human team can unpack
Can you give an example of this?
For the interested; the saying I believe you are referencing in regards to writing code / debugging is from Brian Kernighan, specifically:
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?
It's prescient.
> AI is now capable of constructions so complex that no human or human team can unpack.
How can we possibly know this when we haven't even seriously started on the endeavor of actively reverse engineering these AI-generated proofs? That's a proper job for human mathematicians, because the AIs themselves are demonstrably clueless about what steps in a proof are genuinely interesting and load-bearing from a human POV. This is evidence of a limitation in AIs' capabilities, not of any kind of misaligned behavior. The fact that Tao actually uses that term in his complaint is deeply disappointing.