Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.
https://maxxbench.unlap.ai/research/why-maxxbench/
"My AI is lazy" is honestly not a problem I thought I would have two years ago.
"My AI is lazy" is honestly not a problem I thought I would have two years ago.