Models are trained on ridiculous amounts of human-written text, but their outputs are pretty easy to recognize as not human written. Pangram explains some of the reasons for this in a blog post.[1]

The problem with using Pangram in a feedback loop is that Pangram only reports whether it thinks the text is written by AI or not. It doesn't report whether the text makes sense, or whether it contains errors in spelling or grammar. So when people try to pair AIs with Pangram, the quality of the output tends to degrade to the point where it's not useful.

Basically, it's a harder problem to get an AI to accomplish some goal (such as writing a blog post) while also having its output mimic human text. To use an analogy, it's like building a stealth aircraft with the same speed, maneuverability, range, and payload as a non-stealthy aircraft. Even with a much larger budget, it's not always feasible.

1. https://www.pangram.com/blog/no-llms-dont-just-mimic-human-t...

That's how the earlier image generation models worked. You had one ai to generate images, and one to understand if it's a fake generated image or a real one (like, you have a real image of a dog, use the ai to generate an image of a dog, and pass both to the ai trying to see if it knows which is which). They both start out bad, but evolve together.

https://en.wikipedia.org/wiki/Generative_adversarial_network

That's what adversarial model training is basically.