Abstract
Many recent robot policies pursue stronger control by using large pretrained vision-language models (VLMs) as the action backbone. We introduce WorldDiT, a unified diffusion transformer architecture that couples action generation with visual world modeling and achieves strong performance without a large pretrained VLM action backbone. During training, a single diffusion transformer generates continuous action chunks and predicts normalized RGB patch targets from future camera frames. Across four LIBERO simulation suites, WorldDiT lies on the reported Pareto frontier for total model parameters and mean success among methods reporting all four suites. These results provide a strong sub-billion-parameter baseline for future scaling studies.
keywords:
Robot learning, robot manipulation, diffusion policies, world-action models, world models
1 Introduction
Many robot policies follow a familiar pattern from language modeling, attaching action generation to a large pretrained model. This pattern brings broad perception and language understanding, but it also makes the contributions of scale and architecture difficult to separate. When a policy contains several billion parameters, strong control may come from the action design, the pretrained backbone, or their combination. We therefore ask whether a unified diffusion transformer architecture can combine continuous action generation with an auxiliary future visual prediction objective and retain strong control performance without using a large pretrained vision-language model as the action backbone.
We pursue this objective with WorldDiT, whose frozen visual and language encoders and trainable robot state encoder condition the shared DiT backbone on recent visual observations, robot state, and a language instruction. During training, the shared backbone learns a continuous flow for a seven-step action chunk and an auxiliary normalized RGB patch target selected from future primary-camera and wrist-camera frames. At inference, the shared backbone generates an action chunk. We execute the first three actions, observe the resulting state, and replan.
Figure 2 shows successful rollouts across four LIBERO suites.
| Frame 1 | Frame 2 | Frame 3 | Frame 4 | |
|---|---|---|---|---|
| LIBERO Spatial |  |  |  |  |
| LIBERO Object |  |  |  |  |
| LIBERO Goal |  |  |  |  |
| LIBERO Long |  |  |  |  |
Figure 1 summarizes the reported parameter and performance comparison across all methods. We evaluate one WorldDiT configuration under the specified training and evaluation protocol, providing a baseline for future scaling studies rather than evidence of scaling behavior.
Within this scope, we contribute a unified diffusion transformer architecture for action generation and future normalized RGB patch prediction, a training and deployment design in which RGB patch prediction supplies supervision but is absent at inference, and reported results that characterize the tradeoff between parameter count and mean success below one billion parameters.
2 Method
2.1 Overview
We introduce WorldDiT, a unified diffusion transformer architecture that learns action generation together with an auxiliary future normalized RGB patch objective. Unlike recent world-action models that rely on a large vision-language model to generate action tokens autoregressively, WorldDiT uses one shared DiT backbone to model continuous robot actions and normalized RGB patches selected from future camera frames.
Given a language instruction and a temporally ordered context of observations from multiple views and robot states, WorldDiT encodes the observed context using frozen visual and language encoders. During training, the shared DiT backbone receives action tokens and normalized RGB patch tokens corrupted with Gaussian noise, selected from future primary-camera and wrist-camera frames, and predicts their flow velocities. Future RGB patch prediction supplies auxiliary supervision during training. Deployment follows the action path directly, keeping RGB patch tokens and the RGB prediction head outside the inference graph.
Figure 3 summarizes this path from the observation window and multimodal tokenization through the shared backbone to final slot supervision.
Let an -step trajectory segment beginning at index be
| (1) |
| (2) |
contains the primary-camera image and wrist-camera image , is the robot state, and is the robot action. A language instruction is shared across the trajectory segment.
The first steps form the observation context. Let denote the final observed step and therefore the current control step. The history available at position is
| (3) |
The corresponding action target is an -step action chunk:
| (4) |
We construct the future normalized RGB patch target from the primary-camera and wrist-camera frames at temporal offset . Each CLIP-preprocessed by RGB frame is divided into by patches. Each -dimensional patch vector is normalized using its own mean and variance, and evenly spaced patches are retained from each camera. Concatenating both camera targets gives
The loss is applied only to the final temporal slot, whose RGB patch target comes from the frame at .
Before entering WorldDiT, a trainable linear input projection maps each corrupted -dimensional RGB patch vector to the backbone hidden dimension, and a trainable RGB output projection maps each corresponding hidden representation back to a -dimensional patch velocity. These projections define the model interface and do not change the supervision target, which remains the normalized RGB patch vector. For the robot state, we encode the arm and gripper components separately, concatenate their embeddings, and project the result into one state token for each temporal slot.
2.2 Training
We train the model using flow matching [lipman2023flowmatching]. The WorldDiT backbone receives the clean context tokens, action tokens and normalized RGB patch tokens corrupted with Gaussian noise, their timestep embeddings, and learned register tokens, and predicts the corresponding velocities.
The training objective regresses the velocity of a straight path from Gaussian noise to a clean target. For either an action target or a future RGB patch target, let , denote the clean target, sample , and define
The flow-matching objective is
Here is the context for the current step and is the target velocity.
WorldDiT predicts one velocity for each target. The total loss is the weighted sum of the action velocity loss and the RGB patch velocity loss.
| (5) |
The coefficients and are the corresponding loss weights.
WorldDiT uses the observed steps for conditioning, then predicts one -step action chunk and one future world target composed of normalized RGB patches from the primary-camera and wrist-camera frames at . Only the final temporal slot contributes to the loss.
2.3 Inference
At deployment, the unified backbone receives a -step history ending at the current control step :
| (6) |
The same multimodal encoding path used during training maps to the conditioning representation . No future RGB patch targets or future action labels are provided.
Figure 4 summarizes this deployment path from observed history through action generation and receding-horizon control.
Action generation begins from Gaussian noise:
| (7) |
The learned action velocity field is then numerically integrated:
| (8) |
After integration steps, the final flow state defines the predicted action chunk:
| (9) |
We execute a prefix of , update the history, and replan.
3 Experiments
3.1 Setup
3.1.1 Data and model configuration
We evaluate WorldDiT on the LIBERO manipulation benchmark [liu2023libero]. We use the four downstream suites libero_spatial, libero_object, libero_goal, and libero_10, reported as LIBERO Long, together with the large multi-task libero_90 split used only for pretraining. We convert raw demonstrations to fixed-length windows containing multi-view RGB observations, robot states, language instructions, and action sequences. Future primary-camera and wrist-camera frames in each window provide the auxiliary normalized RGB patch targets.
Visual observations, language instructions, and robot states are encoded by a frozen MAE image encoder, a shared Perceiver Resampler, a frozen CLIP text encoder, and a trainable robot state encoder. The resulting tokens are concatenated and used to condition WorldDiT.
The WorldDiT backbone has depth , hidden size , attention heads, and register tokens. Each input to the unified backbone contains a temporally ordered context of observations. The unified backbone predicts an action chunk, , and uses normalized RGB patch tokens from each future camera frame, for target tokens in total. The complete training window contains steps. We train this configuration solely with the unified flow-matching objective of Section 2.2.
3.1.2 Training
On LIBERO we first pretrain on the multi-task libero_90 split for epochs with a per-GPU batch size of and gradient accumulation of , a learning rate of with a cosine schedule and one warmup epoch, sequence length , observation window , a context length of observations and a complete data-window length of steps.
For LIBERO we fine-tune the pretrained libero_90 checkpoint independently on each downstream suite. Fine-tuning uses an effective batch size of , with a per-GPU batch size of and two gradient accumulation steps on eight GPUs. The unified head is trained on action and normalized RGB patch targets, with action and RGB patch loss weights of and respectively, with a learning rate of in bf16 mixed precision.
We conduct all training runs on one node with eight RTX Pro 6000 GPUs.
3.1.3 Evaluation
Under this protocol, we evaluate one WorldDiT configuration, fine-tuned separately for each of four suites, over episodes per suite.
At inference there is no clean target. We initialize the action tokens from Gaussian noise and integrate the predicted velocity field from to with Euler steps, , decoding the resulting action chunk and sliding the context window. All simulator rollouts use headless EGL rendering. On LIBERO we evaluate each suite with action ensembling and report success rates for each task and their mean over the suite’s tasks. The unified backbone predicts seven actions. We execute three actions before replanning and apply temporal ensembling to overlapping absolute-time action predictions.
For each suite, the reported WorldDiT score aggregates simulator episodes. The aggregate includes episodes per suite used during staged checkpoint selection. It is therefore not fully held out, and the reported should not be interpreted as an unbiased test estimate. The incomplete public availability of model weights and training code prevents us from reproducing every compared method under a shared protocol. We therefore use scores from the cited reports. The graph and table use total instantiated parameter counts, including frozen modules required at inference, and label reconstructed counts as approximate.
3.2 Results
We evaluate a separately fine-tuned WorldDiT model on each of the four LIBERO suites over simulator episodes. WorldDiT records on Spatial, on Object, on Goal, and on Long, producing a mean success rate of . Long remains the hardest suite, which is consistent with the extended multistage behavior required by its tasks.
Figure 1 plots the same rows against total instantiated parameter count. Seer is shown at its reported Long score but is excluded from the Pareto calculation because a mean over all four suites is unavailable.
| Category | Method | Spatial | Object | Goal | Long | Mean |
|---|---|---|---|---|---|---|
| Large pretrained VLM action backbone | ACoT VLA [zhong2026acot] | 98.6 | 99.0 | 99.4 | 97.0 | 98.5 |
| MMaDA VLA [liu2026mmada] | 98.8 | 99.8 | 98.0 | 95.2 | 98.0 | |
| VLANeXt [wu2026vlanext] | 99.0 | 99.2 | 96.6 | 94.8 | 97.4 | |
| VLA Adapter [wang2026vlaadapter] | 97.8 | 99.2 | 97.2 | 95.0 | 97.3 | |
| OpenVLA OFT [kim2025oft] | 97.6 | 98.4 | 97.9 | 94.5 | 97.1 | |
| [physicalintelligence2025pi05] | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 | |
| MemoryVLA [shi2026memoryvla] | 98.4 | 98.4 | 96.4 | 93.4 | 96.7 | |
| Discrete Diffusion VLA [liang2025ddvla] | 97.2 | 98.6 | 97.4 | 92.0 | 96.3 | |
| VLA 0 [goyal2025vla0] | 97.0 | 97.8 | 96.2 | 87.6 | 94.7 | |
| [black2024pi0] | 96.8 | 98.8 | 95.8 | 85.2 | 94.2 | |
| GR00T N1 [bjorck2025groot] | 94.4 | 97.6 | 93.0 | 90.6 | 93.9 | |
| UD VLA [chen2026udvla] | 94.1 | 95.7 | 91.2 | 89.6 | 92.7 | |
| FAST [PertschK-RSS-25] | 96.4 | 96.8 | 88.6 | 60.2 | 85.5 | |
| CoT VLA [zhao2025cotvla] | 87.5 | 91.6 | 87.6 | 69.0 | 83.9 | |
| WorldVLA [cen2025worldvla] | 85.6 | 89.0 | 82.6 | 59.0 | 79.1 | |
| SpatialVLA [qu2025spatialvla] | 88.2 | 89.9 | 78.6 | 55.5 | 78.1 | |
| OpenVLA [kim2025openvla] | 84.7 | 88.4 | 79.2 | 53.7 | 76.5 | |
| No large pretrained VLM action backbone | WorldDiT | 98.0 | 97.0 | 92.8 | 91.8 | 94.9 |
| DreamVLA [dreamvla] | 97.5 | 94.0 | 89.5 | 89.5 | 92.6 | |
| FlowVLA [zhong2025flowvla] | 93.2 | 95.0 | 91.6 | 72.6 | 88.1 | |
| DiT Policy [hou2024ditpolicy] | 84.2 | 96.3 | 85.4 | 63.8 | 82.4 | |
| Octo [octo2024] | 78.9 | 85.7 | 84.6 | 51.1 | 75.1 | |
| Diffusion Policy [chi2023diffusionpolicy] | 78.3 | 92.5 | 68.3 | 50.5 | 72.4 | |
| Seer [seer] | N/A | N/A | N/A | 87.7 | N/A |
WorldDiT uses million total parameters, including frozen modules required at inference, and million trainable parameters. This point lies on the Pareto frontier in the reported comparison. Among methods with a reported mean over all four suites, every method with a higher reported success value uses more total parameters, while every other method at or below the WorldDiT parameter count reports lower success. This characterizes the tradeoff between parameter count and reported mean success among the published results included in the comparison.
4 Discussion
WorldDiT shows that a single diffusion transformer can couple continuous action generation with future visual prediction while retaining an action-only deployment path. Its reported LIBERO performance places the 399M-parameter system on the parameter–success Pareto frontier, indicating that strong benchmark performance is attainable without placing a large pretrained vision-language model in the action backbone. More broadly, this result supports unified world-and-action modeling as a compact basis for scaling across model capacity, data diversity, and deployment settings. Future work can characterize how the normalized future RGB objective shapes control and whether the same tradeoff persists at larger scales. Since action and visual targets share a flow-matching formulation, WorldDiT may also support decomposition into independently trained experts, following the direction explored in Paris and Paris 2.0, including configurations suited to heterogeneous hardware and compute-constrained robot platforms [jiang2025paris, rouzbayani2026paris2].