> Most models tried pleasing an imagined grader over the prompt

>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.

I ran into the same problem on my English exams.