Models are trained on ridiculous amounts of human-written text, but their outputs are pretty easy to recognize as not human written. Pangram explains some of the reasons for this in a blog post.[1]

The problem with using Pangram in a feedback loop is that Pangram only reports whether it thinks the text is written by AI or not. It doesn't report whether the text makes sense, or whether it contains errors in spelling or grammar. So when people try to pair AIs with Pangram, the quality of the output tends to degrade to the point where it's not useful.

Basically, it's a harder problem to get an AI to accomplish some goal (such as writing a blog post) while also having its output mimic human text. To use an analogy, it's like building a stealth aircraft with the same speed, maneuverability, range, and payload as a non-stealthy aircraft. Even with a much larger budget, it's not always feasible.

1. https://www.pangram.com/blog/no-llms-dont-just-mimic-human-t...