I recommend people look at the actual study and think about how representative are the subjects, the tasks involved (SAT essay writing), and the way LLMs are being used.

https://arxiv.org/abs/2506.08872

To be concrete, this is taking a task in isolation that LLMs can do much better than humans (writing garbage essays) and using LLMs to do that task. In the real world, tasks have parts and they exist in a larger context. When we use LLMs for one part of a task, there are other things we're doing that the LLM is not helping with. If you compared people doing arithmetic by hand and with a calculator, you would also see very big differences in how active their brains are. But it's not anyone's job to add up numbers. Adding up numbers is a subtask of a subtask in someone's job.