alt Hacker NewsWhat exactly does world model mean? Ive seen it used so many times in so many ways to just describe SOTA anything its lost its meaning.
Replies
It's an overloaded term for AI models that have spatial reasoning LLMs currently lack.
Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.
There's a bunch of different approaches people are trying:
- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)
- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)
- Google is betting on generative video
- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions
It means everything to everyone.
However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.
The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.
There are other meanings. but essentially a world model is able to reason in 3d, rather than text.
It means it builds internal representation of the world it understands (can do physics on/predict/modify) and then renders it.
It's an overloaded term that different people use for different things. At World Labs we think about three major categories of world models:
- Renderers output pixels or views of a world, either the real world or a generated world
- Simulators predict how the world responds to actions
- Planners predict actions for an agent to make to affect some change in the world
Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.
Source: https://www.worldlabs.ai/blog/taxonomy-of-world-models