July 14, 2026, (Inside AI) — A new system from MIT CSAIL and Toyota Research Institute uses three AI agents to build detailed 3D virtual environments, giving robots a richer training ground and cutting real-world testing time.
Dubbed SceneSmith, the framework deploys a designer, critic, and orchestrator—each powered by the vision-language model GPT-5.2—to collaboratively assemble realistic indoor scenes from text prompts. The result: digital playgrounds with up to six times more objects than prior methods, enabling robots to practice manipulation tasks like placing fruit on plates or moving a soda can before ever touching a physical object.
The work directly tackles a stubborn bottleneck in robotics: the scarcity of diverse, high-quality training data. Physical data collection is slow and expensive, while simulated environments often lack the clutter and variety of the real world. “One of the remaining challenges has been creating sufficiently rich and diverse simulation content to capture the complexity of the real world,” said Russ Tedrake, MIT professor and CSAIL principal investigator.
SceneSmith’s three agents operate in a loop. The designer VLM proposes a layout; the critic VLM flags unrealistic elements—like a bathtub in a living room; the orchestrator VLM decides when the scene is finished. This back-and-forth produces floor plans, furniture, wall objects, and articulated items such as cabinets that robots can open. Once the visual scene is set, physics properties are added via simulation software.
In experiments, the team generated over 1,300 scenes. A pretrained robot policy—trained on real-world data and never exposed to SceneSmith—successfully executed commands like “take the apple from the bowl and place it onto the cutting board” inside the generated environments. That suggests the virtual spaces closely mirror real settings. The team also teleoperated robots through the scenes, opening cabinets and navigating rooms, confirming the environments hold up under physical interaction.
“We’ve found that the system can construct 3D scenes the way a human designer would,” said Nicholas Pfaff, MIT PhD student and lead author. “It made insanely creative and diverse arrangements. I hadn’t taught the system to do that in the prompts; it just improvised.”
SceneSmith also proved useful for policy evaluation. When a VLM agent assessed robot action plans across 100 unique scenes, it identified failures that humans agreed with over 99% of the time. This could help engineers weed out flawed approaches in simulation before real-world deployment.
In a user study with more than 200 participants, SceneSmith’s visuals were rated more realistic over 90% of the time compared to baselines like HSM and Holodeck. It also followed prompts more faithfully, generating requested spaces such as a private office, a pottery store, or a Minecraft-themed gaming room.
The system can even generate individual 3D objects from scratch. A prompt like “a rolling serving cart” produces a 2D image that is converted into a model with mass, friction, and inertia. However, the multi-agent scrutiny comes at a cost: generating a single scene can take hours. The team expects efficiency gains with more compute and hopes to eventually handle deformable objects like sponges.
Jeremy Binagia, an applied scientist at Amazon Robotics not involved in the work, called SceneSmith “a significant advance” for its agentic framework, object density, physical accuracy, and ability to generate assets beyond a fixed library.
The research was presented as a spotlight at last week’s International Conference on Machine Learning. Co-authors include Thomas Cohn, Sergey Zakharov, and Rick Cory. Support came from Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation.
