alt Hacker NewsMy concern with most of these visual benchmarks, popular as they are, is that they are likely more indicative of knowledge (i.e. how comprehensive the training data is and how well it can be retrieved from the model) than of reasoning ability. I don't see in particular how a model would construct a CoT that mapped somehow to a representation of the cube geometry and its animations in latent space without a large chunk of that being pre-existing information.
> without a large chunk of that being pre-existing information.
Is there any evidence that novel reasoning is present in LLM? I've never been able to make that work, and I believe Apple's paper some time ago was good evidence that it doesn't exist. In my experience, sparse latent spaces result in a complete, comical, failure in reasoning.