World Labs
World Labs, the startup founded by AI pioneer Fei-Fei Li, has unveiled a simulation engine that trains robot control systems entirely in virtual environments. The models then run reliably for hours on real hardware.
The company's "Real-to-Sim-to-Real" (R2S2R) engine turns real-world robot tasks into simulations for training and evaluating control models, cutting out expensive tests on actual hardware. The technology comes from SceniX, a startup World Labs acquired in July.
The main bottleneck in robot deployment isn't model architecture, World Labs says, but the sheer volume of experience a robot needs to operate reliably. Real-world data is expensive and hard to control, and even online videos don't systematically cover the full range of objects, physical conditions, and failure states.
One real-world task becomes thousands of controlled variations
The engine captures robots, sensors, the environment, and task demos, then rebuilds them as an interactive virtual world that doesn't just look like the original but behaves the same way physically. World Labs pulls this off by combining generative world models with task-oriented robot simulation.
From a single real-world task, the system generates thousands of variations by changing lighting, object position and count, the surrounding environment, physical properties like friction, and camera angle. To check accuracy, World Labs runs the same action sequence in simulation and reality side by side and compares observations, object movements, and outcomes.
The examples shown include rigid, movable, and deformable objects such as cable routing, inserting an elastic cable end into a hole, and packing a box with both hands.
Control models that never trained on real hardware
Control models train in simulation and then transfer to real robots. One of the test platforms was ALOHA, an open-source dual-arm design from Stanford operated through puppeteering with two smaller control arms. The setup costs a fraction of commercial systems, and all blueprints are public, making ALOHA the go-to reference platform in robotics research.
According to World Labs, the models each ran for one hour across four additional robot platforms without any human intervention. Tasks ranged from wrapping a power cord around a refrigerator with both hands to precisely repositioning test tubes and separating thin objects like markers or pencils from a dense jumble.
The system isn't tied to a specific control model or robot type, so a world that's been reconstructed once can be reused later for new models and different robots, the company says.
Simulation can stand in for hardware during policy evaluation
World Labs argues that robot development lags far behind language models because evaluating control models has mostly required tests on real hardware. A solid simulation doesn't need to deliver the same success rates as reality. What matters is whether it answers the same questions: Where does a model fail, which version is better, and do improvements carry over to the physical robot?
The team tested this with a two-handed cube handoff between the arms of an ALOHA robot. According to World Labs, the simulation reproduced both borderline cases where the robot barely grasped the cube by its edge and the matching failed attempts. Across different model types, including GR00T N1.6 and π₀.₅, and training stages, model rankings in simulation and reality stayed largely the same. That held for both known cube positions and previously unseen ones. Each checkpoint was evaluated using 2,000 simulated and 100 real runs.
Development teams can filter out weak model versions in simulation and save expensive hardware tests for the most promising candidates.
The simulator sits at the center of World Labs' broader strategy
World Labs ties these results back to its taxonomy of world models. In that framework, the simulator is the central piece because it turns a world into a place where software agents can act, learn, and be tested. The company draws parallels with autonomous driving, where some successful Level 3 and Level 4 systems train on a mix of real and simulated data.
World Labs frames its long-term goal this way: to scale the intelligence of robots, you have to scale the worlds in which they learn. How well these results transfer to more complex environments, other robot types, and less controlled everyday situations remains an open question.
World Labs was founded in 2024 by AI researcher Fei-Fei Li to build models with spatial intelligence that understand the three-dimensional physical world. An early system generated walkable 3D environments from single photos. More recently, the company raised one billion dollars in venture capital to extend its world models into robotics and science. The R2S2R engine is the first concrete application of that vision in robotics.
This research feeds into a broader debate about the role world models should play in robotics. An international research team recently tried to pin down a uniform definition of what a world model actually is, drawing a clear line between world models and pure video generators.
A related field is World Action Models, which tie predictions about the near future directly to control commands. That differs from the World Labs approach, where simulation and policy stay separate. Another method called Orca comes from China and lets a robot learn tasks purely by watching video, with no real motion data needed during training.
Read on for the full picture.
Subscribe for hype-free coverage.
- Full access to every article on THE DECODER
- No ads
- Join the comments and community discussions
- A weekly AI news recap via mail
- 6x/year: "AI Radar" — deep dives on the AI topics that matter most
- Daily AI news, always up to date
- Our full ten-year archive
- Covered by a team with 10+ years in AI