AI cannot explain chess moves it comes up with in an elegant way. What makes you think it will be able to do so for math?
It's kind of astonishing that after all we have seen in the last years people still find the position that AI will not be able to do an obviously valuable thing likely and it requiring an explanation (instead of the other way around).
Interesting claim. That's true for old models that simply have no way to explain, LLMs however can. [1]
[1] https://dev.to/natcher/researchers-develop-method-to-train-l...