I agree, LLMs in general tend to misuse words and concepts like "proof" - when they have only partially or sloppily carried out an (heuristic) argument, or in situations where a proof is not possible at all, only empirical evidence. This has improved with recent models but has not been entirely solved