logoalt Hacker News

teraflopyesterday at 11:36 PM2 repliesview on HN

The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.

If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.

For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.

There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.


Replies

JohnnyAPItoday at 12:53 AM

As a World Labs customer, I suspect they are using existing 3D assets when creating their environments. For one thing, a vendor of 3d models for AI mentioned Word Labs as a customer on a webinar.

I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?

iandanforthtoday at 12:41 AM

What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.