Which you have sent to Anthropic/OpenAI/Google's servers when you run the benchmarks for the previous models.

Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.

It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.