Theoretically, you can’t benchmaxx ARC-AGI, but I too am suspect of such a large improvement, especially since the improvement on other benchmarks is not of the same order.

Benchmark maxing could be interpreted as benchmarks actually being a design framework? I'm sure there are pitfalls to this, but it's not necessarily bad either.

Francois Chollet accuses the big labs of targeting the benchmark, yes. It is benchmaxxed.

I assume all the frontier models are benchmaxxing, so it would make sense