I think it could indicate that "semi-private" dataset likely leaked to their training data.

ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra

tl;dr it's 62% when apples-to-apples to other models, which is still notable.

The no-reasoning version scores 35% while the low reasoning one scores 17%? What?

Look at those costs!

But scored less on V2 and V1 ... too much overfitting?

saturated before (higher degree) AGI-2