BAAI's Orca world model matches the performance of specialized systems across five robotics tasks, even though its base model was trained without a single action label. The approach could help solve robotics' chronic data shortage.

The Beijing Academy of Artificial Intelligence (BAAI) has released Orca, a "world foundation model" that breaks from the dominant approach in today's AI. Instead of predicting the next token, the next video frame, or the next robot action, Orca models the next state of the world in an abstract internal representation.

According to the technical report, intelligence shouldn't be defined by specialized prediction models like language models, video generators, or robot controllers. What's needed, the team argues, are models that build a general grasp of how the world changes and can use that as a base for many different tasks.

Two training methods that work together

Orca combines two learning modes. "Unconscious learning" uses raw videos without any captions. The model sees an image and predicts what the next one will look like, not at the pixel level, but in an abstract space, picking up motion patterns, occlusions, and typical scene dynamics along the way.

"Conscious learning" adds verbal instructions. Videos are split into segments, each labeled with a description of the state change, so the model learns how a state shifts when a specific action happens. It also trains on classic video question-answering tasks so it can still respond to natural language queries.

A frozen core with swappable output heads

The pre-trained language-image model Qwen3.5 serves as the base, and after training, this core stays frozen. For each output type, the researchers attach a separate, smaller module that converts the internal state into the desired form. Text runs through Qwen3.5's existing language head. For images, Stable Diffusion 3.5 stays unchanged too, with only small upstream adapters trained to pass Orca's internal state to the image generator. Robot actions come from a control module trained from scratch called "Action Expert."

The split is intentional because the team says that it isn't trying to chase top scores on single benchmarks. Instead, they want to show that a well-trained internal world state can serve as a shared base for very different tasks.

For training, the researchers put together 125,000 hours of video footage, 160 million event descriptions, and 11.5 million question-answer pairs. The videos span four views, covering first-person shots of everyday interactions, third-person shots of object handling, robot recordings without action data, and naturally occurring scenes. Only one-tenth of the video data went into the current version.

Scaling behavior and benchmark results

Orca was trained at two sizes, with 0.8 and 4 billion parameters. Training loss drops steadily with more data and larger models, and early tests also show that the better the internal world state gets during pre-training, the better results are across all three output modes.

On text benchmarks, Orca-4B posts the best overall average of 51.8 percent across MVBench, TemporalBench, 3DSRBench, and SWITCH among the compared small VLMs and larger world models. It beats several VLM baselines like Qwen3.5-4B, Gemma 4-4B, and DeepSeek-VL2-3B on average, though it doesn't win every single test since Qwen3.5-4B leads on MVBench, for instance. Orca-4B also tops the much larger world models Emu3 (8B) and Emu3.5 (34B) on average.

For image prediction, the researchers built their own benchmark called PRICE-V0.1, which uses real-world robot and first-person scenes and asks the model to generate an image showing the result of a command like "close the microwave." Orca-4B hits 59.8 percent on average, beating specialized image generators like FLUX.2 small (56.1 percent), FLUX.1-context (40.9 percent), and OmniGen2 (39.6 percent). Orca keeps the robot's shape, its contact points with objects, and its link to the instruction intact better than pure image models, which often add irrelevant objects or hallucinated hands.

Robot control without action data during pre-training

In five manipulation tasks using a two-armed humanoid robot on wheels, including shelving books, stacking bowls, and scooping sugar, Orca matches π0.5, a system built specifically on robot data. Orca's base model never saw during pre-training which movement goes with which image. For actual robot control, a separate module was trained afterward on 200 real-world recordings per task, pairing camera images with the movements performed.

V-JEPA 2.1 and Qwen3.5, paired with the same control module as baselines, fall well behind. Orca also shows an edge in error recovery, retrying after a failed grasp while π0.5 stays stuck in repeated failures according to the examples in the paper. The authors see this as a potential way to ease robotics' chronic shortage of labeled action data.

What's still missing

For training, the team uses its in-house FlagScale library with several memory and communication tweaks, reaching 2.91 training samples per second per GPU on H100 cards. That's about 4.4 times faster than StarVLA, a codebase widely used in robotics.

Orca still only learns from images and text, with sound, force, and touch missing entirely. Visual prediction runs in the space of a pre-trained image encoder rather than learning its own world space from scratch, and at 0.8 and 4 billion parameters, the models are too small for full world modeling. The event descriptions cover only short windows of a few minutes. BAAI says a native world model trained from scratch on many signal types remains the end goal.

How world models should even be defined is still debated in AI research. A team led by Peking University proposed a unified definition with OpenWorldLib that rules out pure text-to-video models like Sora. A Tsinghua University benchmark showed that even Sora 2 and Veo 3.1 struggle with physically plausible scene progression. In robotics, World Action Models combine state transition prediction with action generation, making it possible to learn from ordinary everyday videos that contain no action data at all.