A very fast librarian with a select material of reference books and adept at speed reading could give the same answers as LLMs.

This is called the "Chinese Room" argument, postulating that a human equipped with a Borgesian library of reference books in a language he doesn't understand and a symbolic lookup table for that language can emulate a thinking human mind without actually thinking.

It was controversial for decades, and is now known not to hold up at all. (Or to hold up perfectly, if unlike Searle your goal is to show that human minds are nothing all that special.) To the extent that the room's occupant succeeds at a task that requires thinking, he is thinking... end of story. Simulated intelligence is now known, thanks to LLMs, to be indistinguishable from real intelligence.

Ask the operator of a Chinese room, who doesn't know Chinese or math, for a novel proof of an unsolved conjecture. The LLM can give you one, but your hypothetical reference librarian won't even know what to look up at first. By the time they learn, the core premise of the argument will no longer hold true.