Atlas: A World Model for Spatial Intelligence

The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.

If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.

For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.

There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.

This is incredible. One potential application that I'm thinking about already is the rapid iteration of video-game map blocking. Being able to drop in some 'initial state' configuration and then have it procedurally generate a handful of alternative configurations could make rapid prototyping a significantly quicker experience, especially if you wanted to see what a potential end result could look like.

Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.

I'm very excited about AI tooling moving forward if this is a glimpse into the future.

What exactly does world model mean? Ive seen it used so many times in so many ways to just describe SOTA anything its lost its meaning.

This seems like by far the best model yet for reconstructing 3D spaces from sparse images. It looks like you could reconstruct your whole house with pretty good fidelity from a dozen or so images taken on your phone.

They show it working with videos that have motion, but it seems like time is always frozen while the camera is moving, and they always return to a ground truth camera view before advancing time again. Maybe the temporal consistency isn't very good? This surprises me given how well it understands space. I guess modeling physics and time is the next step in the development of this kind of model.

I'm a cofounder at World Labs - happy to answer questions about Atlas!

How fast can it generate a frame? If its fast enough for real time then you wouldn't need to pipe the outputs to a different novel view synthesis method like Gaussian splats.

> For robotics, reconstruction is only half the job: as a simulated robot moves through space, Atlas also generates the RGB and depth data its sensors would observe along the way. The world and the robot's view of it come from the same model.

Potentially very significant for accelerating the data flywheel challenge for robotics

Can a reconstruct distinct areas of a larger area with a "fog of war" in between, or will it hallucinate the unknown areas?

A camera moving through a 3D space the world model understands is getting much closer to real robotics applications

It's a promising approach - and the demo goes to show just how advanced and robust "3D from 2D" reconstruction is now.

Dedicated depth sensors used to be a must on advanced robotics platforms - the only way to get anything close to reliable 3D point clouds was to spin a LiDAR. But by now, I wouldn't be surprised to see more and more robots ship with smartphone-like camera blocks - varying FoVs and focal depths, but not a lot of explicit depth sensing, if any at all.

Also, I wonder if this very model can be retrofit into a true robotics VLA? If it already takes text and image guidance, performs autoregressive diffusion of novel views, and handles temporal dynamics - why not diffusion of actions too?

Any ideas if it keeps continuity - when you turn around, is the invented chair still there, or does everything shift like in a dream?

"Can reconstruct [scenes from Unreal Engine]"

[flagged]