alt Hacker NewsLLMs have shown a way to absorb large amounts of textual training data and be useful. For sensory processing you need a different architecture to absorb spatio-temporal streams. Bolting vision onto language models is stretching the tech a bit too thin.
The good news is the data is plentiful — it is all around (the robot). The real world — the robot’s umwelt sensed through its sensors and acted on by its actuators — will provide heaps of data.
Interestingly my colleagues propose that embodied learning will lead to grounding, alignment, and AI safety.
(Worked for 4 years at a robotics + machine learning startup, Director of R&D.)
Replies
Why would the tech be "stretched too thin"? We have made it work better by adding images, video, audio. Why would other sensory input be any different?
Embodiment for general purpose goals like "grounding, alignment, and AI safety" just seems to be one of those things that some people want to be true. There's no actual reason for it to be true. There are no findings showing it to be true. People just want it to be true anyway.
I'm very skeptical of that. Wishful thinking is a treacherous thing.
LLMs provide some incredibly solid "general purpose commonsense reasoning" primitives. And the likes of Astra, Opus 5.5 and Gemini 4 have demonstrated generational leaps in "spatial problem solving" - including robotics tasks. Those don't even have dedicated "sensor input embedding" and "action output diffusion" ends rolled into them at model level - but you can give them control over robot arms, and they'll have a decent chance of, slowly, finding a way to complete new tasks you'd previously would have to tune a robotics model for.
Previously, I have considered "generative video models" to be a decent candidate for the foundation of future robotics models - they, by necessity, perceive video, learn the kind of world dynamics required to plan and execute actions in real world environments. And they can begin to learn those dynamics off embodiment-free general purpose web-scale datasets.
But non-robotics LLM generalists have come to show they're no slouches either. They're not at all optimal for things like rapid precise manipulation. But they find more and more use in robotics pipelines anyway. Makes me wonder what something like "Astra reasoning engine but with robotics architectural extensions" could accomplish.