I think this is fine as long as the success criteria are strict enough that they are forced to learn something in the end.

Any significant testing will inevitably create that situation. The LLM won't have enough context to handle the more precise business requirements. The dev will have to read the code carefully and make their changes by hand. Additional rounds of testing may cause thrashing between regressed states until something clicks for the developer. That lightbulb going off is called "learning" and they are human after all!