You'd just move from bugs in code to bugs in the spec.

Real software needs mind blowingly complex specifications and while it's now fairly easy to get an AI generated proof that your code follows the spec, there's not much to help you with bugs in the spec.

AI will happily change the spec for you so the code verifies though, but that doesn't solve the trust issue.

Truly a trusting trust scenario.