Mathematicians provide two complementary services bundled together.

1. Proving theorems - what AI can apparently replicate faster and better.

2. Creating definitions and new theorems from those definitions to prove, selecting which of the possible statements to work on. I.e. developing the "language" of mathematics. So far there is no evidence that LLM can do this at all well. And there's some reason to think that mathematicians won't be as good at this if they aren't also doing the first part.

The value to society only comes when they do both "well", and it's 2 which is really the black magic where we don't understand why they've been so useful to us.