You are assuming that getting the AI to generate correct (or accurate, representative) data will be very difficult. I would agree, but I think it will become possible in the long run.

(Edit: alternatively you just use AI to get rid of the need for data to solve a problem, like Jev did for traditional classification models)

I think current incentives definitely go against any efforts to build this. It's very hard to build this and be rewarded for it by, say, investors or your boss, because you can't really prove that your system is non-sloppy while your competitor's is (even if being non-sloppy is all that matters), because by definition your novel results are not verifiable or else the model labs will have already trained it into their model.

But the same is true for high-quality AI systems in general. In general, I think AI model advancements will make the systems easier and easier to build until some small guy accountable to no one but themselvs can build it, and then it will actually be built.