> LLMs are still very weak at spatial reasoning compared to diffusion models
Don't know what diffusion model can do, but 100% agree with the "LLMS are very weak at spatial reasoning" comment.
I build a rather complex blueprint-image-to-3D-brep-model a couple of months back using codex ... ugh the damn thing has really no idea where things are in space, something a 3 year old figures out instinctively.
It did end up saving some time as compared to modeling the object myself in a CAD package, but there was so many completely obvious thing I had to explain ... very hard to believe when compared to what codex can pull of with code.
This sounds like a cool project, I would love to hear more about it. I am trying to solve a similar problem myself.