These models aren't really LLMs -- they don't just operate on text tokens. They often include vision models and in some cases audio models. That means that they can better associate the meaning of images and words together so that when someone says "make this button blue" or "create a 3D model of a rocket" they have some level of understanding of what that is and what needs to be done.

The key question is how good that understanding is. For example, a model would likely have a good understanding of various named colours and hex values (e.g. from the HTML specs, X11 specs, and various colour comparison websites) such that it could reasonably correlate that to a CSS entry. It's not clear if/how well a model would identify that given an image, though it should be easy to generate a dataset of image to colour name and/or hex code for training and evaluation.

What's more interesting is whether these frontier models are at their core transformer models, whether they use residual streams to facilitate learning, and whether they are using some other as yet unpublished architecture that gives them an edge.

The coordinate system in game scenes is known as "world space", but that rarely means our world.

How do you train an LLM to create a world that only exists in an artist's head?

I think spending a day with just the lighting systems alone would alleviate us of any misunderstandings here. Getting lighting to work right isn't something you can solve by duct taping a vision model to the contraption.