This is not true. Think of the problems AI has with creating appetizing food imagery, for example. Images that every human annotator marks as unappealing with a high degree of consensus.

The problem is in the rewards, but it's more how they average across domains and contexts, so AI misapplies them. Actually, in human preference evaluations, most of Gen Z in particular extremely adverse to the common look of AI images.

So it's not the annotators involved in RLHF, it's the models failing to map to human preferences at the task level. The failure rate is also uneven across domains.

Where there are challenges with human annotation, it's not the ability to gauge what's good but rather how to correct what's wrong. Most people can tell when something is off but struggle to articulate why. That's where more domain expertise is required.

The saddest words of tongue or pen: “you’re not the target demographic”

I still can’t believe the amount of deepfakes and slop I see just everywhere including food menus and people don’t even seem to notice.

This kind of content is here to stay and we’re going to see more and more of it. Unfortunately…

No, you are confusing a model with reality.