> Use your own data if you want to prove or refute what was seen in my corpus.

I don't think you understand. What you posted is highly dependent on your corpus. I can't "refute" anything because it's not available and it's the major variable in the experiment.

> The point of the benchmarks to establish a baseline for what thinking-token counts one should expect from specific effort levels using published, not what every response should be delivered. If you looked closer, you would see that P90 invocations were still delivered 13x thinking tokens below that level.

I think you're missing something from that first sentence, but I assume you're talking about the comparison to ARC-AGI-2 published thinking tokens?

It should be blindingly obvious that you do not want your thinking token counts to be as high as a benchmark that was designed to push LLMs to their limit.

> Only -46- out of 36,374 July and August invocations even broke 16k thinking tokens - only 0.13% of the total. You are welcome to present what percentile you think is a fair comparison to make here.

What point are you even trying to make?

Again, you don't want invocations to be burning 16K thinking tokens except for rare problems that 1) must be solved in one step and 2) are designed to be entirely self-contained thinking in that step.

You're trying to compare development work to a benchmark that encapsulates complex thinking into a single step.

Coding work is iterative and works in incremental steps: It runs commands, reads more files, checks the web. Thinking tokens should be low for your turns.

ARC-AGI problems have an input and an output. They look like this: https://arcprize.org/tasks/b5ca7ac4 They have more thinking tokens because that's the entire state. They get one output and it's constrained.

> But if you want to knock a month of someone's time, do it with your own data to at least help move the conversation forward.

It is fair to discuss a published analysis. Saying that only people who bring their own month of equivalent analysis (which conveniently would take another month to produce) are allowed to critique it is just a cheap trick to shut people down.

If you post big claims, they are open to analysis and review by others