This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.
Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.
I got the feeling it was too large and inefficient at the time.