We don’t trillions of dollars in LLM investment to build things mathematicians don’t understand. We already have plenty of those, even from ancient times.

As to your second point, Terry Tao already has an answer [1]: the proof isn’t the contribution, shared understanding is. This issue was already raised back when the four-colour theorem was proved. Machine proving and machine proof checking are useful tools but they don’t mean anything without the interpretative work and the communication necessary to build shared understanding.

[1] https://news.ycombinator.com/item?id=49056620

The entire point of writing proofs is for advancing human understanding. A giant dump of symbols that passes the lean compiler is meaningless besides human beings understanding it.

"The age of humans comprehending things is coming to an end"

That's something AI companies would really want you to believe.

You’re prescribing elegance to a stochastic generator trained on the wealth of humanity, including 4chan. Let’s set our expectations a bit.

> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.

I agree.

> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.

I don't know if I see this being true for quite a while, if ever.

> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.

That sentiment makes me cringe. If you understand how LLMs work, you'd know it'll never be possible without a fundamental change in how these work.

We're also supposed to be reaching that point, somehow, without the LLMs ever being intelligent (in the dictionary definition sense, not the "high reasoning model" marketing sense).

Based on observations, the ones who are fooled by the supposed emergent properties, are just that, fools. Any sufficiently unintelligent agent will perceive transformer based LLM text predictors as possessing high intelligence.

In limited experimentation: AI will certainly make statements that are extremely intricate and hard to understand, in part because they're overcomplicated and in part because they use a bunch of unnecessary terminology.

This is not to say that a human couldn't understand a streamlined version or that the AI would not be better if it made more streamlined statements to begin with.

(I am not saying that everything mathematical that an AI produces is in any sense trivial.)

Then it's pretty bad that LLMs don't understand anything.

They don't know and can't know. Without an external source of input that corrects them, their output can never be verified.

It’s possible, but there’s a difference between vastness and difficulty.

Humans can’t compete with AIs on vastness of material they are familiar with, or the depth of effort they are willing and able to throw at a problem.

But scale isn’t the only aspect of difficult scientific endeavours. There’s also theory. And advancements sometimes come through hard graft of knotting together many things. And sometimes they come through the revelation of a deeper truth, or a new framework, a fundamental insight.

AI might help us reach the next level. But that doesn’t mean we won’t understand anything. It could be we have periods of vast intricacy we cannot follow, punctuated by profound elegance we (or at least experts) relatively easily can. And then the scaffolding we needed to get there falls away.

You could be right, but you're making a lot of assumptions about how complexity, scientific understanding, and explanations scale. One of the features of a good scientific discovery is that it often simplifies and compresses things that were previously a bunch of scattered facts. Also, as AI systems improve they'll get better not only at making scientific discoveries, but also at producing understandable explanations.

> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.

Ask yourself: is that really the world you want to live in? It's a world where people, all people, are sidelined.

I think the happy ending of that path is something like Idiocracy. And the more likely ending is something like "automated capitalist economy without the people, because the people couldn't compete."

If and only if that is actually true, then perhaps nothing matters. Until then, calling out shenanigans remains a noble art.

That maybe true at some point, but i don't think we are there yet.

But apparently we can teach machines to do it for us

It's going to produce proofs far more intricate than humans can understand

The thing is, some number of advanced proofs start out "too intricate for most mathematicians to understand" but many of these get rephrase and reframed until they're accessible to undergraduates. Hopefully, AI math can be guided to do that sort of reframing to increase the level of accessible math as well as extend the border of math.

>produce proofs far more intricate than humans can understand

Math is not magic, a proof is just a series of applications of a set of rules on some axioms. A mathematician could understand any proof given enough time to study it; the only way for AI to make proofs that a human couldn't understand is by making really, really long proofs.

Let’s not jump the gun. Where they are right now they can somewhat match our abilities. We haven’t even gotten to the point where they can self improve.

Why would you want something you don't comprehend? How can you be sure it empowers you?

I think perfect rationality doesn't exist, because it is rational to reject something that you don't understand. So rationality of a given physical system will always be bounded.