Who says you need to? You can't definitively prove that prior art does/doesn't exist, either. That's not an impediment to getting a patent. The patent will be examined and issued based on the evidence found. If invalidating evidence is found later, the patent can be invalidated.

In a legal context what is necessary is evidence, not a math/logic formal proof.

> Also, if AI generated content cannot be copyrighted, they can't infringe copyright as well

Because AI cannot pay fines, go to jail, or be assigned the rights of a human. However, a human who uses AI can. If you use AI to infringe copyright, you have infringed the copyright, not the AI.