蚂蚁集团旗下具身智能团队 Robbyant 发布了灵宝 VA 2.0，这是首个原生具身基础模型。该模型被描述为面向通用机器人操控的视频-动作基础模型。研究团队对整个技术栈进行了面向具身智能的预训练，而非对视频生成器进行微调。
什么是灵宝 VA 2.0？
大多数视频-动作模型会复用两个为数字内容创作而构建的组件：一个是面向重建的 VAE，另一个是双向视频扩散主干网络，并附加一个动作模块。
这带来了三个局限性：像素重建潜变量保留了外观信息，但携带的物理结构信息有限；对视频 token 进行迭代式去噪对于闭环控制而言速度过慢；通用的视频训练目标从未教会模型动作如何改变世界。
第四个不匹配是结构性的：主干网络使用双向注意力机制，而控制过程严格按时间正向展开。灵宝 VA 1.0 版本通过微调将该技术栈改造为因果模型，而 2.0 版本则原生预训练了一个因果 DiT。
版本一：语义视觉-动作分词器
基于上述动机，第一阶段替换了仅用于压缩的 VAE。遵循 RepWAM 的思路，该分词器在重建目标之外增加了两个训练目标。
语义对齐将视觉潜变量拉向一个冻结的感知编码器教师模型；潜变量-动作目标提取相邻潜变量之间的紧凑转移变量；逆动力学模型预测每个潜变量对应的动作；前向动力学模型将其解码为传输映射与残差。
世界状态与动作现在共享同一个潜空间。因此，无标注的网络视频也能携带与动作相关的监督信号。
版本二：带有稀疏 MoE 视频流的因果 DiT
在该潜空间之上，版本二预训练了一个因果 DiT。它保留了 1.0 版本的混合 Transformer 架构布局：一个视频专家与一个动作专家共享同一套因果自注意力机制，各自拥有独立的前馈通路。
两条流路的规模呈非对称扩展。视频专家将其密集的前馈网络替换为稀疏的 MoE 路由层。该层包含 128 个路由 SwiGLU 专家、top-8 路由机制以及一个共享专家。负载均衡采用无辅助损失的 Loss-Free Balancing 策略。动作专家则保持密集前馈网络，隐藏维度为 768。
视频主干网络参数约 130 亿，其中约 19 亿为激活参数。加上动作专家和 MCP 头，训练覆盖约 153 亿参数。推理时每个 token 约激活 25 亿参数。训练采用整流流目标函数，配合混合 Muon 加 AdamW 优化器。
训练信号从何而来
除了架构设计，两个目标函数塑造了模型的学习内容。
多块预测（MCP）解决了短视监督问题。教师强制方法仅监督下一个块，导致模型可通过复制外观来降低损失。MCP 附加了三个轻量级模块，用于预测后续三个块。在消融实验中，它在 2 万步内达到了基线模型 4.5 万步的准确率，实现了 2.3 倍的训练加速。
与此同时，五个目标函数采用联合训练而非分阶段训练：文本到图像、文本到视频、文本图像到视频动作、上下文学习以及人机协同训练。采样遵循从粗到细的调度策略，从外观基础学习逐步过渡到视频动作控制。保持所有目标函数活跃，可避免遗忘早期先验知识。
分层规划
块级控制无法对长期目标进行序列化编排。因此，在策略层之上设置了一个 VLM 规划器，该规划器采用冻结视觉塔的 LoRA 微调方案。它输出结构化的 JSON 数据：完成状态、指令、生成指令、局部场景描述。该规划器以约 2Hz 的频率运行，背后依托异步共享缓冲区。策略层在每个块边界读取该数据，因此规划器延迟不会阻塞执行。
前瞻推理
即便采用稀疏主干网络，部署时仍会遇到串行瓶颈。如果机器人等待，模型延迟就会变成控制延迟。
因此，前瞻推理将预测与执行作为异步流运行。当机器人执行块 a_t 时，视频专家会想象其执行结果。动作专家则据此解码出 a_{t+1}。
向前运行存在漂移风险。因此，每次观测返回的结果都会被编码到真实潜在变量 z_{t+1} 中，覆盖掉想象出来的那个。一个前向动力学接地损失函数负责训练视频专家以执行此角色。
# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).
# Not runnable: policy, executor and encode() are placeholders.
C = init_kv_cache(encode(obs_0)) # feedback-grounded cache C_t
a = policy.action_expert(C) # cold start: first action chunk a_0
while not done:
executor.start(a) # execution stream, non-blocking
C_tmp = C + [a] # prediction stream: C_t u {a_t}
z_hat = policy.video_expert(C_tmp) # forward dynamics -> imagined z_{t+1}
a_next = policy.action_expert(C_tmp + [z_hat])
obs = executor.wait() # real observation of a_t returns
C = overwrite(C_tmp, z_hat, encode(obs)) # re-ground: z_hat <- true z_{t+1}
a = a_next
性能表现
因此，评估涵盖了仿真环境和真实硬件。在 RoboTwin 2.0 上，每个模型都在 2,500 个干净演示和 25,000 个随机化演示上进行训练，涉及 50 个任务。
| 方法 | 干净 | 随机化 | 平均 |
| X-VLA | 72.9 | 72.8 | 72.9 |
| π0.5 | 82.7 | 76.8 | 79.8 |
| Motus | 88.7 | 87.0 | 87.9 |
| LingBot-VA | 92.9 | 91.6 | 92.2 |
| LingBot-VA 2.0 | 93.8 | 93.4 | 93.6 |
| 加速技术 | 推理时间（毫秒/块） | 异步频率（Hz） |
| BF16 PyTorch 异步 rollout 基线 | 927 | 35 |
| + 一致性蒸馏 | 466 | 69 |
| + 低精度编译执行 | 369 | 87 |
| + 长程注意力优化 | 272 | 118 |
| + 运行时开销减少 | 142 | 225 |
蒸馏将视频采样器从 5 步减少到 2 步，将动作采样器从 10 步减少到 2 步。FP8 TensorRT 引擎、带有 FlashInfer 注意力的分页/不规则 KV 缓存，以及主机端开销的消除，提供了其余的性能提升。
# Reproduces Table 3 of the report exactly. Runnable as-is.
K = 32 # low-level control steps inside one generated chunk
stack = [("BF16 PyTorch async rollout baseline", 927),
("+ Consistency distillation", 466),
("+ Low-precision compiled execution", 369),
("+ Long-horizon attention optimization", 272),
("+ Runtime overhead reduction", 142)]
for name, ms in stack:
print(f"{name:40s} {ms:4d} ms {round(1000 / ms * K):4d} Hz")
print("end-to-end speedup:", round(927 / 142, 1), "x")
版本 1.0 与版本 2.0 对比
| 维度 | LingBot-VA | LingBot-VA 2.0 |
| 分词器 | Wan2.2 VAE（重建） | 语义视觉-动作分词器，96 个潜在通道 |
| 骨干网络来源 | 从双向生成器微调而来 | 从头预训练的因果 DiT |
| 视频 FFN | 密集 | 稀疏 MoE，128 个专家，top-8 |
| 额外监督 | 未使用 | MCP、上下文学习、人机协同训练 |
| 推理 | 异步执行，KV 缓存 | 带有观测重接地的预见性推理 |
| 峰值异步控制 | 版本 2.0 报告中未报告 | 225 Hz |
分词器消融实验单独列出了第一行。将 WAN2.2 VAE 替换为语义分词器，使一个 1.3B 模型从 78.0 提升到 86.6。
用例与示例
除了基准测试之外，有四种部署形态尤为突出。
- 少样本快速上手：报告指出，该模型可从 10 到 15 个演示中适应。实际评估中每个任务使用 20 个遥操作演示。一个多任务检查点覆盖了所有四个评估任务。
- 演示条件控制：上下文学习允许用人类演示视频替代文本指令。在四个已见任务上微调后，该策略执行了未见过的组合任务。一个例子是：“把葫芦放进绿色盘子里。”
- 廉价数据扩展：人机协同训练将手部姿态重新映射到机器人动作空间。每只手都成为一个虚拟平行夹爪。以自我为中心的语料库包含 6.54 万个片段。
- 反应式控制：演示包括空气曲棍球和传送带场景，其中策略能够预判移动中的物体。
关键要点
- 从头开始预训练一个因果视频动作 DiT，而非适配现有的视频生成器。
- 语义分词器将世界状态和潜在动作置于同一对齐空间中。
- 稀疏 MoE 视频流：约 153 亿参数中每个 token 激活约 25 亿参数。
- 前瞻推理将预测与执行重叠，并在每次真实观测时重新锚定。
- 分块延迟从 927 毫秒降至 142 毫秒；异步控制从 35 Hz 提升至 225 Hz。
交互式动态解释器
Robbyant, the embodied AI unit inside Ant Group, has released the LingBot-VA 2.0.The first embodied-native foundation model. It describes a video-action foundation model for generalist robot manipulation. The research team pretrains the whole stack for embodiment instead of fine-tuning a video generator.
What is LingBot-VA 2.0?
Most video-action models reuse two components built for digital content creation. One is a reconstruction-oriented VAE. The other is a bidirectional video-diffusion backbone, with an action module attached.
This creates three limitations. Pixel-reconstruction latents preserve appearance but carry limited physical structure. Iterative denoising over video tokens is too slow for closed-loop control. Generic video objectives never teach how actions reshape the world.
A fourth mismatch is structural. Backbones use bidirectional attention, while control unfolds strictly forward in time. LingBot VA Version 1.0 finetuned that stack into a causal model. Version 2.0 pretrains a causal DiT natively.
Version 1: The Semantic Visual-Action Tokenizer
Building on that motivation, stage one replaces the compression-only VAE. Following RepWAM, the tokenizer adds two objectives to reconstruction.
Semantic alignment pulls visual latents toward a frozen Perception Encoder teacher. A latent-action objective extracts compact transition variables between consecutive latents. An inverse dynamics model predicts each latent action. A forward dynamics model decodes it into a transport map plus residual.
World states and actions now share one latent space. Unlabeled web video therefore carries action-relevant supervision.
Version 2: A Causal DiT With a Sparse MoE Video Stream
On top of that space, version 2 pretrains a causal DiT. It keeps the Mixture-of-Transformers layout of version 1.0. A video expert and an action expert share one causal self-attention. Each owns a separate feed-forward pathway.
The two streams scale asymmetrically. The video expert replaces its dense FFN with a sparse MoE routed layer. That layer holds 128 routed SwiGLU experts, top-8 routing, one shared expert. Load balancing follows the auxiliary-loss-free Loss-Free Balancing strategy. The action expert keeps a dense FFN at hidden dimension 768.
The video backbone is roughly 13.0B parameters, about 1.9B active. With the action expert and MCP heads, training covers about 15.3B parameters. Roughly 2.5B activate per token at inference. Training uses a rectified-flow objective with a hybrid Muon plus AdamW optimizer.
Where the Training Signal Comes From
Beyond architecture, two objectives shape what the model learns.
Multi-chunk prediction (MCP) fixes myopic supervision. Teacher forcing supervises only the next chunk, so the model can cut loss by copying appearance. MCP attaches three lightweight modules predicting the next three chunks. In ablation it matched the baseline’s 45k-step accuracy in 20k steps, a 2.3x training speedup.
Meanwhile, five objectives are co-trained rather than staged: T2I, T2V, TI2VA, ICL, and human-robot co-training. Sampling follows a coarse-to-fine schedule, from appearance grounding to video-action control. Keeping every objective alive avoids forgetting the earlier priors.
Hierarchical Planning
Chunk-level control cannot sequence long-horizon goals. Above the policy therefore sits a VLM planner, LoRA-finetuned with a frozen vision tower. It emits structured JSON: done, instruction, generation_instruction, local_scene_description. It runs at about 2 Hz behind an asynchronous shared buffer. The policy reads it at each chunk boundary, so planner latency never blocks execution.
Foresight Reasoning
Even with a sparse backbone, deployment hits a serial bottleneck. If the robot waits, model latency becomes control latency.
Foresight Reasoning therefore runs prediction and execution as asynchronous streams. While the robot executes chunk a_t, the video expert imagines its outcome. The action expert decodes a_{t+1} from that.
Running ahead risks drift. So each returning observation is encoded into the true latent z_{t+1}, overwriting the imagined one. A forward-dynamics grounding loss trains the video expert for this role.
# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).
# Not runnable: policy, executor and encode() are placeholders.
C = init_kv_cache(encode(obs_0)) # feedback-grounded cache C_t
a = policy.action_expert(C) # cold start: first action chunk a_0
while not done:
executor.start(a) # execution stream, non-blocking
C_tmp = C + [a] # prediction stream: C_t u {a_t}
z_hat = policy.video_expert(C_tmp) # forward dynamics -> imagined z_{t+1}
a_next = policy.action_expert(C_tmp + [z_hat])
obs = executor.wait() # real observation of a_t returns
C = overwrite(C_tmp, z_hat, encode(obs)) # re-ground: z_hat <- true z_{t+1}
a = a_next
Performance
Consequently, evaluation covers simulation and real hardware. On RoboTwin 2.0, every model trains on 2,500 clean plus 25,000 randomized demonstrations, across 50 tasks.
| Method | Clean | Randomized | Avg. |
| X-VLA | 72.9 | 72.8 | 72.9 |
| π0.5 | 82.7 | 76.8 | 79.8 |
| Motus | 88.7 | 87.0 | 87.9 |
| LingBot-VA | 92.9 | 91.6 | 92.2 |
| LingBot-VA 2.0 | 93.8 | 93.4 | 93.6 |
| Acceleration technique | Inference time (ms/chunk) | Async Hz |
| BF16 PyTorch async rollout baseline | 927 | 35 |
| + Consistency distillation | 466 | 69 |
| + Low-precision compiled execution | 369 | 87 |
| + Long-horizon attention optimization | 272 | 118 |
| + Runtime overhead reduction | 142 | 225 |
Distillation cuts the video sampler from 5 steps to 2, and the action sampler from 10 to 2. FP8 TensorRT engines, a paged/ragged KV cache with FlashInfer attention, and host-side overhead removal supply the rest.
# Reproduces Table 3 of the report exactly. Runnable as-is.
K = 32 # low-level control steps inside one generated chunk
stack = [("BF16 PyTorch async rollout baseline", 927),
("+ Consistency distillation", 466),
("+ Low-precision compiled execution", 369),
("+ Long-horizon attention optimization", 272),
("+ Runtime overhead reduction", 142)]
for name, ms in stack:
print(f"{name:40s} {ms:4d} ms {round(1000 / ms * K):4d} Hz")
print("end-to-end speedup:", round(927 / 142, 1), "x")
Version 1.0 vs Version 2.0
| Dimension | LingBot-VA | LingBot-VA 2.0 |
| Tokenizer | Wan2.2 VAE (reconstruction) | Semantic visual-action tokenizer, 96 latent channels |
| Backbone origin | Finetuned from a bidirectional generator | Causal DiT pretrained from scratch |
| Video FFN | Dense | Sparse MoE, 128 experts, top-8 |
| Extra supervision | Not used | MCP, in-context learning, human-robot co-training |
| Inference | Async execution, KV cache | Foresight Reasoning with observation re-grounding |
| Peak async control | Not reported in the version 2.0 report | 225 Hz |
The tokenizer ablation isolates row one. Swapping the WAN2.2 VAE for the semantic tokenizer lifts a 1.3B model from 78.0 to 86.6.
Use Cases and Examples
Beyond benchmarks, four deployment shapes stand out.
- Few-shot onboarding: The report states the model adapts from 10 to 15 demonstrations. Real-world evaluation uses 20 teleoperated demos per task. One multi-task checkpoint covers all four evaluated tasks.
- Demonstration-conditioned control: In-context learning lets a human demonstration video replace the text instruction. After finetuning on four seen tasks, the policy executed unseen compositions. One example: “put the calabash into the green plate.”
- Cheap data scaling: Human-robot co-training retargets hand poses into the robot action space. Each hand becomes a virtual parallel gripper. The egocentric corpus spans 65.4k episodes.
- Reactive control: Demonstrations include Air Hockey and a conveyor belt, where the policy anticipates moving objects.
Key Takeaways
- Pretrains a causal video-action DiT from scratch instead of adapting a video generator.
- A semantic tokenizer puts world states and latent actions in one aligned space.
- Sparse MoE video stream: ~2.5B of ~15.3B parameters activate per token.
- Foresight Reasoning overlaps prediction with execution, re-grounded on every real observation.
- Chunk latency 927 ms to 142 ms; async control 35 Hz to 225 Hz.