But don't humans make mistakes too? Like are we sure the failure rate of AI with the right checks and bounds is lower than humans, who are flawed machines themselves?

If you need assurances, have a different LLM write the test suite.