An interesting paper from the GPT 3.5 era is "AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably" by Brian Porter & Edouard Machery (2024).

Reinforcement-learning on human preferences does work, to an extent. You get LLM output that is (shallowly) highly appealing to humans. That's certainly not in doubt.

The problem is that writing has purposes aside from making hordes of (indiscriminating, low-taste) humans smile. And these purposes are actively corroded by RL'ing on human preferences.

Generally, AI generated fiction has problems similar to the award-winnning stories listed here - they are full of incoherent, "poetic" metaphors, vague gestures to narrative depth that is never delivered on, and relentless pandering to the cheap seats. They're very much "playing the hits" - saying unchallenging, nice things that no reader would ever disagree on. We discover that racism is wrong, that family is what matters, that you have to remember who you are.

They all feature simple (almost generic) motifs, such as trees that represent life. (The repetitiveness of these motifs becomes apparent when you start reading a bunch of stories. In "The Serpent in the Grove" by Jamir Nazir we get "‘People does change,’ he says. ‘But grove does remember.’". In Back and Forth by Kavyta Kay we get "Nani smiled, sad and certain. 'Trees remember everything, beti. They live long.")

It also might be a window into how literary contests are judged.

The recent crop of prize winners have striking similarities. Largely African-American or Indian authors. Stories involving "big" humanistic issues like the legacy of colonialism/slavery, the importance of remembering one's roots, gender-based violence, and so on, with the implication that the writer is drawing on their lived experience somewhat.

I strongly suspect that the prizes are not solely being awarded based on literary merit. Judges of these contests fundamentally want to feel like their work matters: that they're uplifting worthy voices. You can deliberately write in a way that flatters this notion (or get ChatGPT to write for you.)