> The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves?
Because OpenAI and Anthropic deliver their most powerful models along with classifiers that stand in the way of potentially rogue actions. This works fine for now (in the HuggingFace incident they had turned it off for testing), but it will keep working if and only if alignment is solved; otherwise an agent swarm might be able to convince the classifier to join.