LEGO-Anything Turns a Single Photo Into an Editable 3D Scene
Researchers at the University of Maryland and Amazon Web Services have introduced a method that looks at a photo and writes code for Blender, the 3D creation software. The appeal is that the resulting scene can be edited, but the AI showed limits in judging how accurate its own output was.
Researchers from the University of Maryland and Amazon Web Services (AWS) unveiled “LEGO-Anything” in a paper published on September 28. The method takes a single photo and turns it into a 3D scene you can edit later. Alongside it, the team introduced “LEGO-Bench,” a suite of tests for measuring how well AI systems build scenes, and “LEGO-Plugin,” which improves the results. The distinguishing feature: the output is not a finished image but a program you can modify and re-run.
Can you actually edit the scenes it builds?
Yes. LEGO-Anything uses an AI that handles coding on its own (an agent) to look at an image and write code for Blender, the 3D creation software. The AI runs the code to render the scene as an image, inspects the result, and then revises the code over multiple passes. Writing code and reviewing the output are both part of the process of converging on a scene that resembles the input image.
In the output, you can change objects, the camera, and more. Unlike approaches that hand you only a finished 3D asset, the individual components of the scene can be edited separately or used for analysis. The output can also support locating objects within a scene, segmenting regions, and extracting depth information that represents distance. That said, in the researchers’ comparisons, performance on these individual tasks fell short of models purpose-built for each one.
Is a top score of 53.4% good?
In the performance tests, GPT-6 Astra took first place in overall score among six models, with 53.4% on indoor scenes and 39.6% on outdoor scenes. The evaluation used 208 images generated from 104 indoor and outdoor simulator scenes. Rather than real photographs, the comparison relied on images of virtual scenes that come with ground-truth information.
The overall score weighs three things: whether the scene file can be used without errors, how accurately the shapes of visible surfaces match, and how closely the re-rendered image resembles the original. Shape and similarity scores are averaged in only when the system produces a usable scene file; outputs that fail that bar are scored as zero. The researchers noted that even among runs that did produce usable scene files, there was wide variation in how well object geometry and appearance matched the original scene.
The team identified three problems. First, the quality of the initial scene. Second, a tendency for the revision process to break parts that were already working. And third, the AI misjudging the accuracy of its own output. Even with the ability to run code and re-examine the results, it struggled to produce accurate geometry.
How much does the plugin help?
LEGO-Plugin requires no additional training. It assists with the initial composition of a scene and supports measurement-grounded revisions and version management of earlier states. In a narrower test limited to 42 office-scene cases, all six models improved their scores. The largest gain was 62.7% above the prior score.
The code is not yet available to download and use. The project page hosts the paper and examples, with the code listed as “coming soon.” What to watch going forward: how accurately the method recovers geometry from real photographs, and whether iterative revision improves results consistently.