alt Hacker NewsDo birds expose enough of their cognition through birdsong?
Do birds expose locomotion-relevant functions specifically through birdsong?
Do we have enough birdsong data available to start solving the inverse problem?
If "yes" on all, then we might be able to do it.
I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That looks an awful lot like a set of two "yes".
The last open question is: is there enough spatial reasoning reflected in the language data we have?
It's plausible that spatial reasoning is too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily. And it's also plausible that existing LLM architecture is uniquely poorly suited to learning spatial reasoning - higher cognitive functions involved in things like writing code or even composing poetry are a better fit for the architecture. And it's plausible that we're underestimating just how complex spatial reasoning truly is - Moravec's paradox strikes again.
We know that LLMs perform poorly and improve slowly on spatial reasoning tasks, but not exactly why. And progress on things like ARC-AGI series shows that they're not completely inept.
I was meaning to imply that yes assuming we had a proportionate amount of birdsong data, would we be able to reverse engineer their flight abilities.
I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily”
I think this is pretty apparent. It’s very rare for athletes to talk through their actions in high level detail - I saw the ball coming towards me at a 37 degree phi 23 degree epsilon angle at a speed of approximately 20 mph, I estimated it’s time to arrival would be .45 seconds etc. The eye-hand coordination occurs almost completely outside of what you consider conscious awareness. And it’s not easy to describe that’s why athletic coaching is difficult to do through words alone.
As far as ARC-AGI goes it looks like last years models were scoring <5% against their v2 benchmark: https://arxiv.org/pdf/2505.11831
Frankly I don’t understand why you can’t train a multi-modal LLM on video game frame data. Is that just way too compute intensive to do? What am I missing here? Because I think it’s crazy to think that an LLM could learn to think spatially just from reading… even if they’re reading everything that’s ever been written. I think that about summarizes my position.