同等贡献。通讯作者：Arnav <arnav-kumar.jain@mila.quebec> 和 Yilin <yilinwu@andrew.cmu.edu>。
摘要
世界模型（WM，即学习型模拟器）对机器人技术的潜在影响十分深远——涵盖策略评估、策略改进以及测试时规划——且所有这些都只需有限的真实世界交互。为了释放这些下游能力，世界模型需要同时满足三个必要条件：(i) 保真度（即生成与真实情况相关的模拟轨迹），(ii) 一致性（即生成长时间跨度内连贯的模拟轨迹），以及 (iii) 效率（即快速生成模拟轨迹）。我们提出 WEAVER（面向具身推理的多视角世界估计）：一种同时满足上述三个必要条件的世界模型架构，在机器人操作任务上取得了最先进的结果。WEAVER 是一个多视角世界模型，通过流匹配损失来预测未来的潜在状态和奖励值。我们提炼了模型架构、记忆和预测目标方面的关键设计决策，这些决策对于攻克此前世界建模方法难以处理的长期动态操作任务至关重要。我们将 WEAVER 应用于机器人硬件，证明了其在策略评估（与真实世界成功率的关联性）、策略改进（在机器人基础模型之上实现真实世界成功率的提升）以及测试时规划（相较于先前世界模型，真实世界成功率提升且速度加快）方面的有效性。在分布外场景下评估时，WEAVER 也展现出优于先前世界模型的性能。代码、模型及视频请访问：https://arnavkj1995.github.io/WEAVER/。
1 引言
世界模型（WM，[13]），即学习型模拟器，已引起学术界 [12, 54, 51, 33] 和工业界 [6, 36] 的浓厚兴趣。这是因为世界模型为机器人技术带来了巨大前景：无需代价高昂且往往不安全的真实世界交互，即可评估和改进策略。此外，当世界模型被整合到规划算法中时，还能实现测试时扩展。
为了同时实现评估、改进和规划这三项承诺，机器人世界模型必须共同满足三个核心需求。首先是（i）保真度：生成与真实世界结果相关的、物理上准确的预测。其次是（ii）一致性：在长预测范围内保持预测的连贯性。第三是（iii）效率：快速生成预测。例如，策略评估和改进需要高保真度的预测（以处理任意的视觉运动机器人策略）以及一致性（以处理多阶段任务）。同样，规划需要快速推理以满足机器人的实时性要求。
尽管进展迅速，但目前尚无机器人世界模型能同时满足所有这三个需求。例如，视频生成模型 [29] 以低效率为代价生成高保真度的内容。同样，JEPA 风格的世界模型 [2] 具有潜在状态，这些状态可能无法解码为评估任意视觉运动机器人策略所需的图像。而尽管 Dreamer-v4 [16] 看起来很有前景，但从头开始学习编码器而非使用预训练模型，可能会损害其在分布外场景下的鲁棒性。
当我们聚焦于机器人操作时，世界建模问题变得更为复杂，因为我们必须处理场景的多视角、从历史信息中推断被遮挡的物体，并确保预测的世界状态具有相对较高的保真度，而不仅仅是视觉上的美观。处理这些复杂性往往以牺牲效率为代价，例如像 Ctrl-World [12] 这样用于操作任务的最先进世界模型，其运行速度远慢于真实世界，这使其无法用于测试时的规划，并使得策略改进在计算上变得困难。
为此，我们提出了 WEAVER（面向具身推理的多视角世界估计）：一种世界模型架构，实现了（i）高保真度、（ii）长程一致性以及（iii）高效生成，从而在具有挑战性的机器人操作任务中，于策略评估、策略改进和测试时规划方面均达到了最先进的性能。为了实现这三重能力，WEAVER 融合了以往世界建模方法中的关键设计决策。从视频生成领域，我们采用了扩散强制（Diffusion Forcing）[7] 和流匹配（Flow Matching）[27]（用于在快速推理速度下实现长程生成），以及使用预训练编码器 [35]（用于提升分布外鲁棒性）。从潜在世界模型 [16, 37, 12] 中，我们采用了奖励预测头，以便无需像 VLM 这样的外部评判模型即可进行高效评估。从 JEPA [3] 中，我们将未来潜在预测（而非图像重建）作为主要训练目标。最后，为了处理机器人操作中的特定复杂性，我们采用了 Ctrl-World [12] 的多视角生成和记忆架构。
综合来看，我们最终形成了一个完整的整体：一个可用于机器人操作的世界模型，能够灵活地应用于评估、改进和规划。在一套包含五项操作任务（从抓取放置到可变形物体操作）的真实硬件实验中，WEAVER 在用于评估时，与真实世界成功率展现出强相关性；在无需任何真实世界交互的情况下，提升了 [21] 机器人基础模型的真实世界成功率；并且实现了比 Ctrl-World [12] 更快的测试时规划。
2 相关工作
机器人世界模型。虽然世界模型已在自动驾驶[36, 45]、视频游戏[18]和代码生成[8]等领域得到探索，但我们重点关注其在机器人技术[49, 12, 37, 42, 3]中的应用——更具体地说，是视觉操作任务。尽管视频生成技术的进步[47, 4]带来了高（i）保真度的世界模型[29, 12, 37, 33, 12, 10, 36]，但这些世界模型通常（iii）效率不足以用于测试时规划。然而，通过融入更广泛视觉社区的关键技术，如流匹配[27]和扩散强制[7]，我们得以提升WEAVER的（iii）效率。此外，使用预训练视频生成模型的编码器[35]增强了WEAVER对分布外视觉输入的鲁棒性，而使用预训练解码器则使我们能够评估任意视觉运动机器人策略，这与JEPA风格的模型[3]不同。最后，我们采用了Dreamer-v4[16]的潜在奖励和价值头，以实现（iii）高效的评估和规划，无需像[11]那样将解码后的图像传递给外部且通常较慢的VLM评判器。
以往的世界模型[14, 15, 17, 49, 50, 23, 22]难以在长时间跨度内保持时间（ii）一致性。为此，我们借鉴了[12, 36]中的多视角预测、历史记录和记忆机制，确保即使在夹爪与物体交互存在遮挡的情况下，生成的序列也能保持连贯。这与WorldGym[33]、DreamerV4[16]和DreamDojo[10]等早期世界模型形成了对比。
与我们的方法最为接近的可能是Ctrl-World[12]和Dreamer-v4[16]。通过采用视频生成社区的技术[7, 27]实现更（iii）高效的推理，我们能够在更短时间内生成更高（i）保真度且时间（ii）一致性更强的序列，在帕累托意义上优于Ctrl-World[12]。通过使用预训练编码器[35]而非像Dreamer-v4[16]那样从头学习一个编码器，我们很可能继承了更好的分布外视觉输入鲁棒性。
世界模型在机器人技术中的应用。世界模型有望在机器人策略评估、改进和测试时规划方面带来“下游”进展。先前的研究表明，足够保真的世界模型能够实现可扩展的策略评估[42, 51, 46]，同时早期结果也提示，合成轨迹可能有助于改进策略[11, 46]，不过其实际效果能达到何种程度仍是一个悬而未决的问题。最近，世界模型被探索用于测试时规划[32, 50]，其核心挑战在于为在线优化快速生成准确结果。WEAVER正是针对机器人操作领域的这些下游应用而设计的。
3 WEAVER：面向具身推理的多视角世界估计
我们现在介绍WEAVER的关键组成部分：一个旨在支持策略评估、策略改进和测试时规划的机器人世界模型。世界模型在操作任务中的这些下游应用，在上游提出了三个关键需求：(i) 在物理交互过程中保持多视角的保真度，(ii) 在可能引入遮挡的长时程交互中保持预测的一致性，以及(iii) 具备足够高效的生成能力，以用于实时规划算法。
为了同时满足这三个需求，WEAVER融合了多种技术要素。我们首先描述关键的世界模型设计决策和训练目标（第3.1节），随后介绍推理加速（第3.2节）和潜在空间价值估计（第3.3节）。最后，我们将展示这些组件如何共同实现评估、改进和规划功能（第3.4节）。
设置：机器人与策略。我们考虑由自然语言指令指定的长时域机器人操作任务。设机器人的本体感知状态（例如关节角度）为 。机器人还拥有场景的 RGB 视图（例如来自腕部摄像头和第三人称摄像头）；设这组多视角图像为 。在时间步 ，机器人同时观察多视角图像和本体感知状态： 。设机器人的动作为 （例如关节速度）。给定任意 ，机器人的基础策略生成一个 步的未来动作块（即 ），随后在 WM/环境中执行。
世界模型架构。我们的 WM 通过一个预训练编码器 将观测映射为潜在状态 。一个关键的设计选择是让我们的世界模型同时以每第 个先验潜在状态的记忆 以及最近 步潜在状态的历史 为条件。给定记忆、历史和一个 步的动作计划 ，WM 预测未来的潜在状态：
| (1) |
其中 是 步的未来。我们还训练了一个奖励模型，用于对预测的潜在状态与语言指令的对齐程度进行评分：其中 。为了实现对视觉运动策略的迭代调用，我们使用一个预训练解码器来获取未来的观测（摄像头视图和本体感知状态），即 ，其中 是 步的未来观测。最终的预测 被反馈给策略，以生成下一个动作块。
3.1 实现高保真、时序一致的世界模型生成的关键设计决策
多视角相机预测。尽管在设计用于操作的视觉运动机器人策略时，使用多个视角（例如腕部相机和外部相机）是常见做法，以应对部分可观测性并捕捉更精细的物体-夹爪交互 [21]，但许多世界模型仅预测单一视角 [33, 10]。遵循 [12, 24, 44] 的做法，WEAVER 同时预测外部相机和腕部相机的观测结果。多视角提供的更多信息有助于通过处理操作过程中的遮挡来提升（ii）一致性。每个视角都使用预训练的 Stable Diffusion 3 VAE 编码器 [9] 编码为图像块 token。我们将本体感受状态投影到相同的 token 维度，并通过拼接图像块 token 和本体感受 token 来获得结果。
本体感受状态预测。除了未来的视觉潜变量，WEAVER 还预测未来的本体感受状态。我们发现，显式预测机器人的配置（而不仅仅是像 Ctrl-World [12] 那样的视觉观测）对于处理可变形物体的接触密集型操作至关重要，因为这类操作通常需要知道手臂的精确位置和夹爪的张开宽度。
稀疏记忆与短期历史。世界模型生成过程中的时间（ii）一致性要求模型理解在交互过程中哪些发生了变化、哪些保持不变。这在操作任务中尤其具有挑战性，因为遮挡和腕部相机视角变化可能导致物体和背景场景的一部分离开或进入机器人的视野。为此，WEAVER 基于 [12] 的方法，在生成未来状态时以两组观测结果为条件：长期稀疏记忆和短期历史。具体来说，记忆包含每第 th 个编码后的观测结果，以帮助捕捉更长期的上下文；而历史则包含最近两帧，以捕捉动作带来的短期后果。
潜在动力学模型。该潜在动力学模型基于记忆、历史记录和候选行动方案来预测未来的潜在状态。为了在（i）保真度与（iii）效率之间取得平衡，WEAVER 采用了遵循文献[16, 33]的高效 2D Transformer 架构，其动力学模块由空间注意力机制和因果时序注意力机制组成。在每个预测步骤中，模型以潜在 token、行动 token 和流时间步嵌入为条件，自回归地生成一个步长的片段。为了稳定训练，每个模块均使用 RMSNorm [52]、RoPE [39]、QKNorm [19] 和 SwiGLU 前馈层 [38]（更多细节见附录 A2）。
训练目标。与文献[16]类似，我们使用流匹配损失 [27] 来训练潜在动力学模型以预测未来的潜在状态。设 表示真实的下一个潜在状态， 表示一个维度相同的高斯噪声向量。接着，我们定义 ，其中 。然后，我们通过最小化均方误差来训练模型预测“速度”：。为了提升长程（ii）一致性，我们采用了扩散强制 [7] 方法，该方法在未来的不同时间步上使用独立采样的噪声水平来训练潜在动力学模型。我们还使用了 SPRINT 模块 [30]，该模块会主动丢弃潜在状态中的补丁 token，以提升（iii）效率。
3.2 加速世界模型推理速度
对于像 WEAVER 这样基于扩散 Transformer 的世界模型 [12, 16, 33]，延迟是（a）模型前向传播和（b）迭代去噪两者的共同结果。因此，（iii）高效生成需要同时解决这两个问题。我们通过在所有去噪步骤中对记忆和历史 token 使用 KV 缓存来降低（a）的成本。我们通过调整去噪过程来降低（b）的成本。具体来说，在扩散强制 [7] 的基础上，我们采用了渐进式噪声调度。与文献[12, 16]中使用的线性调度不同，WEAVER 采用余弦调度以实现更高的（i）保真度生成。
为了进一步提升（iii）效率，使其达到测试时规划所需的水平，我们采用整流流目标[28]对 WEAVER 进行后训练，使其能够在少量前向传播内生成高质量内容。具体而言，我们首先利用去噪过程生成一条高质量的潜在轨迹，然后将其作为二次蒸馏步骤的目标。更多实现细节请参见附录 A2.3。
3.3 基于世界模型的精确高效价值估计
奖励模型。为了实现（iii）对提议动作片段的高效评分，而无需（a）将潜在表示解码为图像以及（b）将其输入外部 VLM 评判模型[11, 33]，我们将现成奖励模型产生的分数蒸馏到一个轻量级奖励头中，该奖励头直接对潜在状态和语言指令进行操作。奖励头使用 AdaPool[5] 聚合潜在 token，随后接入 MLP 层。我们使用简单的均方误差目标进行训练。
评论家。为了支持使用世界模型进行截断时间范围的推演，WEAVER 学习了一个评论家网络，用于估计超出想象时间范围的价值。该评论家与奖励模型共享相同的潜在空间设计，并使用 MSE 目标进行训练，以预测自举的 λ 回报[40]。给定来自 的潜在奖励，目标递归定义为 ， 。然后通过最小化 来训练评论家。
3.4 下游世界模型应用：评估、改进与规划
通过同时满足（i）保真度、（ii）一致性和（iii）效率这三个理想特性，WEAVER 能够支持评估、改进和规划等下游能力。
策略评估。对于策略评估，我们从真实世界推演中获取记录的动作轨迹，在 WEAVER 内部以开环方式执行这些轨迹，并沿途记录预测的奖励值。我们专注于长时域任务，这些任务有时需要对 WEAVER 的潜在动力学模型进行 40 次以上的迭代评估，这凸显了时间（ii）一致性和（iii）效率的重要性。
策略改进。对于策略改进，我们从策略中采样一个长度为 τ 步的动作块，并在世界模型内向前模拟 N 次，总共覆盖 T 个时间步，利用 WEAVER 的（i）保真度和（ii）一致性。从相同的初始观测值出发，重复此过程 K 次后，我们收集到一批轨迹展开。然后，我们计算每条轨迹展开的 τ 步优势函数的蒙特卡洛估计：如果该批次中得分最高的轨迹展开（即 τ*）的优势值超过某个小的正阈值（即 A > δ），我们就将其蒸馏到基础策略中。这种基于优势的过滤机制可以防止策略在那些所有 τ 步采样计划都被预测为比当前策略预期行为更差的状态下被更新[23, 1]。
测试时规划。我们采用单块、最佳 N 次[26]的测试时扩展方法，该方法不涉及迭代调用潜在动力学模型。具体来说，给定当前观测值和指令，我们从策略中采样候选动作块，用世界模型想象它们的结果，并执行通过潜在奖励头和评论家头估算出优势值最高的那个动作块。WEAVER 的（iii）效率（包括潜在动力学模型的速度，以及无需调用外部 VLM 评判器，仅通过奖励头即可评估候选动作序列的能力）对于解锁这种测试时扩展能力至关重要。
4 实验设置
基础策略与硬件。我们的基础策略是[21]，这是一个在 DROID 数据集[41]上训练的最先进的视觉-语言-动作（VLA）策略。我们遵循 DROID 硬件设置，使用一个 Franka Emika Panda 机械臂、两个安装在工作区左右两侧的外部 Zed 2i 摄像头，以及一个安装在手腕上的 Zed Mini 摄像头（见附录图 10）。VLA 策略和我们的 WEAVER 世界模型仅使用右侧摄像头视图和手腕摄像头¹。 ¹我们设置了所有三个摄像头，因为我们的主要世界模型基线[12]使用了全部三个视图。
数据集与任务。为了使世界模型与基础策略的数据分布对齐，我们首先在 DROID 数据集上对 WEAVER 世界模型进行预训练，然后在我们真实世界的设置上对其进行微调。我们通过执行五项真实世界操作任务来收集数据以微调世界模型，每项任务执行 50 次。我们还为每项任务额外收集了 20 次执行数据作为评估数据。我们选择的任务使得基础策略至少能达到一定的成功率，同时涵盖从刚性物体抓取放置到可变形物体操作以及动态操作等一系列能力。具体来说，我们的任务是：叠碗（将一个碗叠在另一个碗上）；抓放袋子（将一个可变形的薯片袋放到盘子上）；抓放记号笔（重新调整记号笔的方向并将其插入杯子中）；抓放毛巾（将一条柔软的毛巾放入篮子中）；以及倒咖啡豆（将装满咖啡豆的杯子倒入碗中）。每项任务的详细信息可在附录 A1.1 中找到。
世界模型训练。WEAVER 是一个拥有 9.28 亿参数的模型。我们在 DROID 数据集 [41] 上使用 32 的批量大小和 1e-4 的学习率在 GPU 上预训练了 100 万步，耗时 10 天。为了在 WEAVER 的潜在表示之上训练奖励模型和评论家，我们使用 Robometer [25] 获得的进度奖励（减 1 以得到负奖励）对 DROID 数据集进行了标注。在世界模型微调期间，模型在我们收集的任务数据上以较低的学习率 1e-5 更新了 1.6 万步。得到的模型用于策略评估、策略微调和测试时规划。与先前的工作 [12] 一样，我们将步长下采样 3 倍，以使用 5Hz 的频率进行世界模型想象。我们将动作表示为两个时间步之间的关节位置差，以匹配策略的动作空间。我们还学习了一个额外的关节速度到位置的动作适配器，用于在数据生成和测试时规划之间转换动作空间（参见 A1.2）。
5 实验结果
我们首先单独研究 WEAVER 世界模型的性能（第 5.1 节），然后研究其在策略评估、策略改进和测试时规划等下游用例中的性能（第 5.2 节）。
5.1 WEAVER 在帕累托最优意义上优于领先的操作世界模型
我们首先将仅在 DROID 数据集上预训练的 WEAVER 与领先的多视角操作世界模型进行性能比较。Ctrl-World [12] 是一个 1.5B 参数的扩散模型，在 DROID 数据集上训练，并从预训练的 SVD 检查点 [4] 初始化。
设置与指标。我们在 DROID 数据集的一个验证集（256 条轨迹）和一个使用 VLA 收集的分布外数据集（100 条轨迹）上评估这两个模型。对于每条轨迹，模型以自回归方式展开，生成 10 秒长的序列，其中每次生成联合预测 15 步动作块（1 秒）的结果。遵循之前的评估 [12]，我们使用 FID [20] 和基于真实视频计算的 FVD [43] 来衡量解码生成的视觉保真度。更多指标详见附录 A3.1。
| 外部视角 | 腕部视角 | 时间 | ||||
| 方法 | NFE | FID | FVD | FID | FVD | （秒） |
| \rowcolorpastellavender DROID（验证集） | ||||||
| Ctrl-World | 16 | 26.09 | 78.73 | 33.83 | 195.37 | 14.65 |
| 50 | 22.44 | 55.05 | 25.32 | 91.77 | 42.33 | |
| WEAVER | 16 | \cellcolorlightblue10.20 | 27.83 | 21.50 | 90.72 | \cellcolorlightblue4.78 |
| 50 | \cellcolorlightblue9.51 | \cellcolorlightblue26.54 | \cellcolorlightblue16.75 | \cellcolorlightblue66.89 | 14.25 | |
| \rowcolorpastelmint 任务数据（分布外） | ||||||
| Ctrl-World | 16 | 36.16 | 139.54 | 38.76 | 277.13 | 14.65 |
| 50 | 31.44 | 91.48 | 33.47 | 145.86 | 42.33 | |
| WEAVER | 16 | \cellcolorlightblue23.95 | \cellcolorlightblue88.27 | 30.77 | 184.62 | \cellcolorlightblue4.78 |
| 50 | \cellcolorlightblue23.48 | \cellcolorlightblue87.03 | \cellcolorlightblue27.37 | \cellcolorlightblue145.04 | 14.25 | |
![[Uncaptioned image]](/api/img-proxy?u=https%3A%2F%2Farxiv.org%2Fhtml%2F2606.13672v2%2Fx3.png&exp=1788156000&sig=9f4987b2c4dc8878bf633408532e2026754e3d4e6d94f39be5cdbfaeb1a4fd52)
结果：感知高保真度生成。表 1 比较了 WEAVER 和 Ctrl-World 在不同评估数据集上的 FID 和 FVD 结果。WEAVER 在推理时间更短的情况下优于 Ctrl-World。随着我们减少函数评估次数（NFE）以降低延迟，我们发现 Ctrl-World 的质量下降幅度比 WEAVER 更显著；两个模型在预测腕部相机视角时均产生最高误差。我们提供了额外结果来比较 NFE（附录 A3.2）、不同噪声调度（附录 A3.4）以及通过 KV 缓存实现的推理加速（附录 A3.3）。
结果：长时域下质量更高。接下来，我们衡量世界模型想象力的感知质量如何受长时域预测的影响。对于两个世界模型，我们生成长动作序列（150 步或 10 秒）的 rollout，并测量每 15 步间隔预测视频的 FID，以估算随时间域变化的生成质量。如图 3 所示，在 DROID 数据集上，我们发现即使推理预算从 50 NFE 减少到 16 NFE，WEAVER 相比 Ctrl-World 仍能保持持续更低的 FID。在 OOD 数据集上，WEAVER 在外景视角上保持了性能差距，并在腕部视角上具有可比性能。
结果：WEAVER 在推理速度与质量上实现帕累托占优。接下来，我们研究在固定推理时间预算（以 NFE 衡量）下，生成质量如何受影响，该预算基于在单个 H100 GPU 上生成 10 秒片段所需的推理时间。在图 5 中，我们看到 WEAVER 在 NFE 为 8、16、32、50 时显著优于 Ctrl-World，同时享有显著更低的推理速度（例如，Ctrl-World 为 30-50 秒，而 WEAVER 为 10-30 秒）。通过帕累托占优 Ctrl-World，WEAVER 实现了更快的评估和规划，我们将在下文第 5.2 节中探讨这一点。
结果：潜在奖励预测准确性。最后，我们将 WEAVER 的潜在奖励预测与 RoboMeter [25] 的奖励标签进行了比较，并在真实保留轨迹上进行了评估。图 4 显示了 PnP Stack 任务一次 rollout 的预测奖励；WEAVER 正确想象了抓取和堆叠等关键事件，且想象奖励与真实 RoboMeter 奖励相关。在图 4 的右侧面板中，我们看到使用预测奖励计算的优势值也能够区分动作样本的不同结果。这是一个有希望的指标，表明 WEAVER 及其潜在奖励适用于第 5.2.1 节中的合成数据过滤和第 5.2.3 节中的测试时规划。
5.2 WEAVER 实现有效的评估、改进与规划
到目前为止，我们已经验证了 WEAVER 有效平衡了（i）保真度、（ii）长程一致性和（iii）高效生成。接下来，我们将转向世界模型的下游应用：策略评估（第 5.2.1 节）、策略改进（第 5.2.2 节）和测试时规划（第 5.2.3 节）。
5.2.1 WEAVER 实现与真实情况紧密相关的有效策略评估
首先，我们评估 WEAVER 是否可以作为离线策略评估的学习型模拟器，从而减少对昂贵的真实世界 rollout 的需求。
设置。给定一个初始真实观测值 和动作序列 ，我们自回归地生成想象观测值，并根据生成的 rollout 来估计策略性能。我们比较了三种世界模型：在 DROID [12] 上预训练的 Ctrl-World、在 DROID 上预训练的 WEAVER，以及在 上微调的 WEAVER-FT。为了测试世界模型在不同基础策略质量下的鲁棒性，我们分别在基础策略和微调策略生成的 rollout 上对每个模型进行了评估。
指标。遵循先前的工作 [51]，我们通过比较人工标注的想象 rollout 上的二值成功率与 上的真实成功率（每个任务平均 20 次试验），来衡量生成 rollout 的性能与真实世界性能的相关程度。我们报告了皮尔逊相关系数 [31] 和最大矩阵排序违反度 (MMRV) [42]（参见附录 A4.1）。
结果。图 6 显示，预训练的世界模型倾向于低估策略性能，但 WEAVER 与真实 rollout 的一致性优于 Ctrl-World，具有更高的皮尔逊相关系数和更低的 MMRV。这种设置具有挑战性，因为 rollout 可能持续长达 40 秒，并且需要准确的长程预测。对于预训练模型来说，倾倒任务尤其困难，这很可能是因为颗粒动力学在 DROID 数据集中代表性不足，且本身难以建模。经过微调后，WEAVER-FT 显著提高了评估准确性，将皮尔逊相关系数提升至 ，并在不同性能的策略之间更好地匹配了真实结果。图 6 左侧的定性示例进一步表明，WEAVER-FT 比基线方法更准确地预测了 PnP Towel 和 Pour Beans 任务的结果。
5.2.2 WEAVER 能够在无需真实交互的情况下实现有效的策略改进
高保真世界模型的另一个理想用途是为策略改进生成合成数据。我们利用世界模型对候选动作片段进行采样和验证，然后将高价值的想象片段提炼回策略中 [23, 1]。
实验设置。为了评估 WEAVER 在改进策略方面的效用，我们探索了多种为策略微调生成数据的策略：(1) 基础策略：在 DROID 上训练的原始 VLA；(2) 使用真实数据微调：我们利用优势估计值修剪真实轨迹中的片段，每个任务得到 1000 个包含 36 步动作块的片段；(3) 使用合成数据微调：我们使用基础策略和 WEAVER 采样多个片段，根据预测的优势值（第 3.4 节）进行筛选，每个任务保留 1000 个片段；(4) 使用混合数据微调：合并经过筛选的真实数据集和合成数据集（每个任务 2000 个片段）（更多细节和结果见附录 A2.2 和 A4.2）。
实验结果。图 7 显示，所有经过微调的策略相比基础策略，成功率均有显著提升。值得注意的是，基于合成数据的微调与基于真实数据的微调结果非常接近，平均性能差距仅为 。这表明我们的合成数据质量极高，能够实现与昂贵的真实世界数据相似的策略改进效果。结合真实数据和合成数据可进一步提升性能，平均成功率比单独使用真实数据微调提高了 。这些结果表明，来自世界模型的想象轨迹为知识蒸馏提供了有价值的来源，减少了对昂贵的真实世界数据采集和人工筛选的需求。图 7 还显示了在接触密集型和动态操作任务上的改进，例如更精确的标记物放置和倒豆子。我们进一步在倒豆子任务上研究了合成数据的规模效应，将想象片段的数量从 1000 个变化到 2000 个和 5000 个。图 7（右）显示，随着合成数据量的增加，策略性能持续提升，最终超过了仅使用真实数据微调所达到的性能。
5.2.3 WEAVER 通过平衡推理速度与质量实现测试时规划
最后，测试时搜索需要在执行前评估多个动作序列，这使得推理速度成为关键瓶颈。与在图像空间中使用重建和 VLM 作为评判者 [11, 37] 进行规划不同，WEAVER 在潜在空间中进行规划，以实现更高的效率 [15, 23]。
实验设置。我们使用基础策略，并采样一批动作块。对于每个动作块，WEAVER 会想象未来状态的潜在表示，并使用奖励头和评论家头评估优势。这降低了解码预测观测结果和查询外部 VLM 评判者的成本。遵循第 5.2.2 节的策略改进设置，我们在五个任务上评估了测试时规划，并与基础策略进行了比较。我们使用并行样本和设定的想象范围，以平衡规划质量和延迟。
结果。我们报告了任务成功率以及测试时规划流程中的推理时间分解。图9显示，基于优势的选择将策略引导至成功的行为。测试时规划将平均成功率提升至基础策略之上，最大提升幅度达到 。当基础策略较弱时，提升幅度更大，但由于规划仅限于单个动作片段且必须在延迟约束下运行，其提升仍小于直接微调。附录A4.3中的表7显示，动力学预测仍是主要的计算瓶颈。尽管如此，在RTX A6000 Ada GPU上，WEAVER比Ctrl-World推理流程[12]快约 ，并且批量采样的扩展速度与候选数量呈次线性关系，这表明我们的推理优化使基于世界模型的测试时规划在实时操控中变得可行。
6 结论
我们提出了WEAVER：一种用于操控的世界模型，它实现了(i)高保真度、(ii)时间连贯性以及(iii)高效生成。在各项任务中，WEAVER与真实世界成功率评估显示出强相关性（），无需任何真实世界交互即可将策略成功率提升 ，并且解锁了比Ctrl-World[12]更快的测试时规划。
局限性。尽管WEAVER释放了大规模世界模型在操控领域的潜力，但仍存在若干局限性。首先，视觉世界模型仅能观测到底层状态的部分视图，可能需要触觉感知来解决歧义。其次，融入物理先验知识可以提升涉及可变形物体操控任务的性能。第三，当前的生成延迟将测试时规划限制在单个动作片段上的短时域推理。最后，来自RoboMeter的奖励监督可能存在噪声，这促使我们开发更好的奖励模型用于失败预测。我们在附录A5中提供了进一步讨论。
更广泛的影响。本研究探索了大规模世界模型，旨在通过减少对昂贵现实世界交互的依赖，提升机器人操作的效率、安全性和可扩展性。想象推演可以在执行前支持策略评估、改进和测试时规划，但不准确或有偏差的预测可能导致风险决策，这在辅助机器人等安全关键领域尤为重要。因此，负责任的部署需要仔细的验证、不确定性估计，以及防范利用所学世界模型或奖励模型中的错误。
致谢
我们感谢 Jesse Zhang 在奖励模型和 ROBOMETER 方面的有益讨论。AJ 获得魁北克研究基金（FRQ）（DOI 编号：https://doi.org/10.69777/350253）、Calcul Quebec 和加拿大卓越研究讲席（CERC）计划的资助。GKS 获得一项 STTR 资助。本研究部分由加拿大数字研究联盟（https://alliancecan.ca）和 Mila（https://mila.quebec）提供的计算资源支持。YW 和 AB 部分获得国家科学基金会（NSF）奖项和 NSF CAREER 奖项的资助。文中观点仅代表作者本人，不一定反映 NSF 的观点。
参考文献
- Anthony 等人 [2017] Thomas Anthony, Zheng Tian, and David Barber. 结合深度学习与树搜索的快慢思考. 收录于《神经信息处理系统大会》（NeurIPS），2017年。
- Assran 等人 [2023] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 基于联合嵌入预测架构的图像自监督学习. 收录于《IEEE/CVF 计算机视觉与模式识别大会》（CVPR），2023年。
- Assran 等人 [2025] Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, 等. V-jepa 2：自监督视频模型实现理解、预测与规划. CoRR, 卷 abs/2506.09985, 2025年。
- Blattmann 等人 [2023] Andreas Blattmann、Tim Dockhorn、Sumith Kulal、Daniel Mendelevitch、Maciej Kilian、Dominik Lorenz、Yam Levi、Zion English、Vikram Voleti、Adam Letts 等。稳定视频扩散：将潜在视频扩散模型扩展到大型数据集。CoRR，abs/2311.15127，2023年。
- Brothers [2025] Greyson Brothers。通过Transformer输出的自适应池化实现鲁棒噪声衰减。国际学习表征会议（ICLR），2025年。
- Bruce 等人 [2024] Jake Bruce、Michael D Dennis、Ashley Edwards、Jack Parker-Holder、Yuge Shi、Edward Hughes、Matthew Lai、Aditi Mavalankar、Richie Steigerwald、Chris Apps 等。Genie：生成式交互环境。国际机器学习大会（ICML），2024年。
- Chen 等人 [2024] Boyuan Chen、Diego Martí Monsó、Yilun Du、Max Simchowitz、Russ Tedrake 和 Vincent Sitzmann。扩散强制：下一token预测与全序列扩散的结合。神经信息处理系统大会（NeurIPS），2024年。
- Copet 等人 [2025] Jade Copet、Quentin Carbonneaux、Gal Cohen、Jonas Gehring、Jacob Kahn、Jannik Kossen、Felix Kreuk、Emily McMilin、Michel Meyer、Yuxiang Wei 等。Cwm：一个用于基于世界模型的代码生成研究的开放权重大语言模型。CoRR，abs/2510.02387，2025年。
- Esser 等人 [2024] Patrick Esser、Sumith Kulal、Andreas Blattmann、Rahim Entezari、Jonas Müller、Harry Saini、Yam Levi、Dominik Lorenz、Axel Sauer、Frederic Boesel 等。扩展整流流Transformer以实现高分辨率图像合成。国际机器学习大会（ICML），2024年。
- Gao 等人 [2026] Shenyuan Gao、William Liang、Kaiyuan Zheng、Ayaan Malik、Seonghyeon Ye、Sihyun Yu、Wei-Cheng Tseng、Yuzhu Dong、Kaichun Mo、Chen-Hsuan Lin 等。Dreamdojo：一个基于大规模人类视频的通用机器人世界模型。CoRR，abs/2602.06949，2026年。
- Guo 等人 [2026a] Yanjiang Guo、Tony Lee、Lucy Xiaoyang Shi、Jianyu Chen、Percy Liang 和 Chelsea Finn。Vlaw：视觉-语言-动作策略与世界模型的迭代协同改进。CoRR，abs/2602.12063，2026a。
- Guo 等人 [2026b] Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, 和 Chelsea Finn。《Ctrl-world：一种用于机器人操作的可控生成式世界模型》。发表于国际学习表征会议（ICLR），2026b。
- Ha 和 Schmidhuber [2018] David Ha 和 Jürgen Schmidhuber。《世界模型》。CoRR，abs/1803.10122，2018。
- Hafner 等人 [2020] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, 和 Mohammad Norouzi。《通过想象控制梦想：通过潜在想象学习行为》。发表于国际学习表征会议（ICLR），2020。
- Hafner 等人 [2021] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, 和 Jimmy Ba。《使用离散世界模型掌握 Atari 游戏》。发表于国际学习表征会议（ICLR），2021。
- Hafner 等人 [2025] Danijar Hafner, Wilson Yan, 和 Timothy Lillicrap。《在可扩展世界模型内部训练智能体》。CoRR，abs/2509.24527，2025。
- Hansen 等人 [2022] Nicklas Hansen, Xiaolong Wang, 和 Hao Su。《用于模型预测控制的时间差分学习》。发表于国际机器学习大会（ICML），2022。
- He 等人 [2025] Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, 等。《Matrix-game 2.0：一个开源实时流式交互世界模型》。CoRR，abs/2508.13009，2025。
- Henry 等人 [2020] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, 和 Yuxuan Chen。《Transformer 的查询键归一化》。发表于自然语言处理实证方法会议（EMNLP），2020。
- Heusel 等人 [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, 和 Sepp Hochreiter。《采用双时间尺度更新规则训练的生成对抗网络收敛于局部纳什均衡》。发表于神经信息处理系统大会（NeurIPS），2017。
- Intelligence 等人 [2025] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, 等。《pi0.5：一个具有开放世界泛化能力的视觉-语言-动作模型》。CoRR，abs/2504.16054，2025。
- Jain 等人 [2022] Arnav Kumar Jain, Shiva Kanth Sujit, Shruti Joshi, Vincent Michalski, Danijar Hafner, 以及 Samira Ebrahimi Kahou。通过变分稀疏门控学习鲁棒动力学。发表于《神经信息处理系统大会》(NeurIPS)，2022 年。
- Jain 等人 [2026] Arnav Kumar Jain, Vibhakar Mohta, Subin Kim, Atiksh Bhardwaj, Juntao Ren, Yunhai Feng, Sanjiban Choudhury, 以及 Gokul Swamy。平静的海面练不出娴熟的水手：通过学会搜索实现鲁棒模仿。发表于《神经信息处理系统大会》(NeurIPS)，2026 年。
- Jiang 等人 [2025] Yuxin Jiang, Shengcong Chen, Siyuan Huang, Liliang Chen, Pengfei Zhou, Yue Liao, Xindong He, Chiming Liu, Hongsheng Li, Maoqing Yao 等。Enerverse-ac：通过动作条件构想具身环境。CoRR，abs/2505.09723，2025 年。
- Liang 等人 [2026] Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, 以及 Jesse Zhang。Robometer：通过轨迹比较扩展通用机器人奖励模型。发表于《机器人学：科学与系统 2026》，2026 年。
- Lightman 等人 [2024] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, 以及 Karl Cobbe。让我们逐步验证。发表于《国际学习表征会议》(ICLR)，2024 年。
- Lipman 等人 [2023] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, 以及 Matt Le。用于生成式建模的流匹配。发表于《国际学习表征会议》(ICLR)，2023 年。
- Liu 等人 [2023] Xingchao Liu, Chengyue Gong, 以及 Qiang Liu。流直且快：通过修正流学习生成和迁移数据。发表于《国际学习表征会议》(ICLR)，2023 年。
- Mei 等人 [2026] Zhiting Mei, Tenny Yin, Ola Shorinwa, Apurva Badithela, Zhonghe Zheng, Joseph Bruno, Madison Bland, Lihan Zha, Asher Hancock, Jaime Fernández Fisac 等。机器人学中的视频生成模型——应用、研究挑战与未来方向。CoRR，abs/2601.07823，2026 年。
- Park 等人 [2026] Dogyun Park, Moayed Haji-Ali, Yanyu Li, Willi Menapace, Sergey Tulyakov, Hyunwoo J Kim, Aliaksandr Siarohin, 和 Anil Kag。Sprint：面向高效扩散 Transformer 的稀疏-密集残差融合。发表于国际学习表征会议 (ICLR)，2026 年。
- Pearson [1920] Karl Pearson。相关性历史笔记。《生物计量学》，13(1):25–45，1920 年。
- Qi 等人 [2026] Han Qi, Haocheng Yin, Aris Zhu, Yilun Du, 和 Heng Yang。通过预测性世界模型在推理时增强生成式机器人策略。IEEE 机器人与自动化快报，11(5):5534–5541，2026 年。
- Quevedo 等人 [2026] Julian Quevedo, Ansh Kumar Sharma, Yixiang Sun, Varad Suryavanshi, Percy Liang, 和 Sherry Yang。Worldgym：将世界模型作为策略评估的环境。发表于国际学习表征会议 (ICLR)，2026 年。
- Radford 等人 [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark 等。从自然语言监督中学习可迁移的视觉模型。发表于国际机器学习大会 (ICML)，2021 年。
- Rombach 等人 [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, 和 Björn Ommer。基于潜在扩散模型的高分辨率图像合成。发表于 IEEE/CVF 计算机视觉与模式识别会议 (CVPR)，2022 年。
- Russell 等人 [2025] Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, 和 Gianluca Corrado。Gaia-2：面向自动驾驶的可控多视角生成式世界模型 (2025)。CoRR，abs/2503.20523，2025 年。
- Sharma 等人 [2026] Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, 和 Sherry Yang。World-gymnast：在世界模型中通过强化学习训练机器人。CoRR，abs/2602.02454，2026 年。
- Shazeer [2020] Noam Shazeer。GLU 变体改进 Transformer。CoRR，abs/2002.05202，2020 年。
- Su 等人 [2024] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, 和 Yunfeng Liu。Roformer：通过旋转位置嵌入增强的 Transformer。《神经计算》，568:127063，2024 年。
- Sutton 等人 [1998] Richard S Sutton、Andrew G Barto 等。《强化学习：导论》，第 1 卷。MIT 出版社剑桥，1998 年。
- Team [2024] DROID 团队。DROID：一个大规模野外机器人操作数据集。载于《机器人学：系统与科学》，2024 年。
- Team 等人 [2025] Gemini 机器人团队、Krzysztof Choromanski、Coline Devin、Yilun Du、Debidatta Dwibedi、Ruiqi Gao、Abhishek Jindal、Thomas Kipf、Sean Kirmani、Isabel Leal 等。在 Veo 世界模拟器中评估 Gemini 机器人策略。CoRR，abs/2512.10675，2025 年。
- Unterthiner 等人 [2018] Thomas Unterthiner、Sjoerd Van Steenkiste、Karol Kurach、Raphael Marinier、Marcin Michalski 和 Sylvain Gelly。迈向精确的视频生成模型：一种新指标与挑战。CoRR，abs/1812.01717，2018 年。
- Wang 等人 [2025] Jianyuan Wang、Minghao Chen、Nikita Karaev、Andrea Vedaldi、Christian Rupprecht 和 David Novotny。VGGT：视觉几何基础 Transformer。载于《IEEE/CVF 计算机视觉与模式识别会议（CVPR）》，2025 年。
- Wang 等人 [2024] Xiaofeng Wang、Zheng Zhu、Guan Huang、Xinze Chen、Jiagang Zhu 和 Jiwen Lu。DriveDreamer：面向真实驾驶的世界模型用于自动驾驶。载于《欧洲计算机视觉会议（ECCV）》，2024 年。
- Wang 等人 [2026] Yixuan Wang、Rhythm Syed、Fangyu Wu、Mengchao Zhang、Aykut Onol、Jose Barreiros、Hooshang Nayyeri、Tony Dear、Huan Zhang 和 Yunzhu Li。用于机器人策略训练与评估的交互式世界模拟器。CoRR，abs/2603.08546，2026 年。
- Wiedemer 等人 [2025] Thaddäus Wiedemer、Yuxuan Li、Paul Vicol、Shixiang Shane Gu、Nick Matarese、Kevin Swersky、Been Kim、Priyank Jaini 和 Robert Geirhos。视频模型是零样本学习者和推理者。CoRR，abs/2509.20328，2025 年。
- Wissler [1905] Clark Wissler。斯皮尔曼相关系数公式。《科学》，22(558):309–311，1905 年。
- Wu 等人 [2023] Philipp Wu、Alejandro Escontrela、Danijar Hafner、Pieter Abbeel 和 Ken Goldberg。Daydreamer：用于物理机器人学习的世界模型。载于《机器人学习会议（CoRL）》，2023 年。
- Wu 等人 [2025] Yilin Wu, Ran Tian, Gokul Swamy, 与 Andrea Bajcsy。从预见走向预谋：基于潜在对齐的 VLM 在环策略引导。发表于《机器人学：科学与系统》（RSS），2025 年。
- Yin 等人 [2026] Tenny Yin, Zhiting Mei, Zhonghe Zheng, Miyu Yamane, David Wang, Jade Sceats, Samuel M Bateman, Lihan Zha, Apurva Badithela, Ola Shorinwa 等。Playworld：从自主游戏中学习机器人世界模型。CoRR，摘要编号 2603.09030，2026 年。
- Zhang 与 Sennrich [2019] Biao Zhang 与 Rico Sennrich。均方根层归一化。发表于《神经信息处理系统大会》（NeurIPS），2019 年。
- Zhang 等人 [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, 与 Oliver Wang。深度特征作为感知度量的惊人有效性。发表于《IEEE/CVF 计算机视觉与模式识别大会》（CVPR），2018 年。
- Zhou 等人 [2025] Gaoyue Zhou, Hengkai Pan, Yann LeCun, 与 Lerrel Pinto。Dino-wm：基于预训练视觉特征的世界模型实现零样本规划。发表于《国际机器学习大会》（ICML），2025 年。
目录
附录 A1 机器人设置与任务
A1.1 任务详情
我们在五个操作任务上收集了真实世界的微调数据，每个任务在我们的 DROID 设置上执行 50 次 rollout，如图 10 所示。我们选择基础策略至少能达到一定成功率的任务，以确保收集的 rollout 中既包含成功执行也包含失败执行，同时仍处于策略的能力范围之内。这些任务旨在覆盖多样化的操作场景，包括刚体拾取与放置、可变形物体操作以及动态操作，如图 10 所示。
叠碗任务要求机器人将一个碗叠放在另一个碗之上。两个碗被随机放置在桌面上，机器人必须将碗 放置在碗 上，其中 和 。
拾放袋子任务要求机器人拿起一袋薯片并将其放在一个绿色盘子上。我们使用两种类型的薯片袋，并在每次试验中随机抽取一种。袋子是可变形物体，这使得抓取结果和物体运动难以预测。
PnP Marker 任务要求机器人拾取水平放置在桌面上的 Expo 白板笔，并将其放入容器内。白板笔的颜色从黑色和橙色中随机选择，目标容器则从纸杯和蓝色马克杯中随机选取。该任务需要精确抓取以及大幅度的末端执行器重新定向，以便将白板笔垂直插入。
PnP Towel 任务要求机器人拾取一条毛巾并将其放入篮子中。我们使用了两种毛巾变体：一条折叠的厚红色厨房毛巾和一条薄灰色方形毛巾，以及两种篮子变体：橙色和蓝色。该任务具有挑战性，因为毛巾是可变形物体，其最终形状很大程度上取决于抓取位置，对于折叠毛巾而言，还取决于抓取的层数。
Pour Beans 任务要求机器人拾取一个装有咖啡豆的杯子，并将其倒入一个蓝色碗中。该任务测试动态操作能力，因为咖啡豆的颗粒运动难以预测，成功执行需要精确控制杯子的姿态、倾倒角度和运动轨迹，以避免咖啡豆洒到碗外。
A1.2 动作空间
DROID 设置上的基础策略输出关节速度指令以进行控制。为了匹配这种动作表示，我们将世界模型的动作空间定义在关节空间中，从而避免将动作转换为笛卡尔空间等其他表示时可能产生的复合误差。然而，我们发现直接以关节速度为条件来训练世界模型会导致生成质量较低。因此，遵循先前的工作[12]，我们使用一个轻量级的动作适配器将关节速度转换为关节位置。
在策略评估中，由于每条轨迹末尾的关节位置已经可用，我们直接使用关节位置作为世界模型的输入。然而，在测试时规划过程中，必须根据策略提出的关节速度动作进行预测。因此，我们使用训练好的动作适配器来预测相应的关节位置，并让世界模型基于这些适配后的动作进行生成。以下章节将详细描述动作适配器的具体细节。
A1.2.1 动作适配器
概述。
动作适配器是一个轻量级前馈模块，用于桥接世界模型的动作表示（关节速度指令和二进制夹爪信号）与机器人的可观测状态（绝对关节位置和夹爪宽度）。给定机器人当前状态以及世界模型生成的一段动作，它能够预测由此产生的关节位置增量和夹爪位置增量序列，随后将这些增量累加得到未来的绝对状态。
输入表示。
模型接收两组输入：
-
状态 token：当前 7 自由度关节位置与当前标量夹爪位置拼接，形成一个 维向量。
-
动作 token：一段关节速度指令（）与夹爪动作（）拼接，展平为 维。
两组输入拼接成一个 维的单一输入向量（对于 的情况）。
架构。
适配器是一个三层 MLP，隐藏层大小为 512，激活函数为 SiLU：
输出被重塑为 ，并拆分为预测的关节增量（）和预测的夹爪增量（）。
归一化。
所有连续输入和目标值均使用训练集计算的每个维度的第 1/第 99 百分位边界进行最小-最大归一化至 ，这种方法相比全局最小/最大值对异常值更鲁棒。夹爪动作指令在输入前被二值化（，否则为 ），以反映其离散的开/关语义。
损失函数。
模型使用加权均方误差损失在归一化的增量目标上进行训练：
夹爪的权重被提高，因为其动态范围远小于关节维度，否则在抓取和放置预测中，相对于其重要性，夹爪的误差会被低估。
推理。
在测试时，模型进行预测，对其进行反归一化，并从当前状态开始进行积分：
训练细节。
该适配器使用 Adam 优化器（批次大小 128）在 50 小时的本体感觉遥操作数据上训练了 15 个周期。每个训练样本包含从某个片段中随机抽取的一段连续时间步窗口；第一个时间步提供当前状态，其余时间步提供动作块和增量目标。
附录 A2 实现细节
A2.1 架构细节
我们使用 Stable Diffusion 3 [9] 中的 VAE 将来自相机视角的 19032 帧图像编码到潜在空间。我们的高效 Transformer 架构是一个 32 层 Transformer，具有 1536 个隐藏维度和 16 个注意力头。每一层包含一个空间层，用于关注所有图像块中的信息，以及一个因果时序层，用于关注来自先前观测的图像块。动作和本体感觉状态使用 DROID 训练数据集获得的统计数据进行归一化。我们从 Robometer [25] 获取奖励标注，并使用进度奖励来训练奖励头和评论家网络。奖励和评论家网络使用一个 AdaPool [5] 层将 token 压缩为向量，并包含多层感知机层，该向量与 CLIP 嵌入 [34]（提供语言指令的表示）拼接在一起。
A2.2 训练细节
WEAVER 预训练。WEAVER 总共有 928M 参数，在 4 块 H100 GPU 上训练了 100 万梯度步，耗时 10 天。预训练在 DROID 数据集 [41] 上进行。在训练过程中，我们还维护了模型权重的指数移动平均（EMA）。我们在初始的 10000 步使用学习率预热，并在预热后保持恒定的学习率。我们在表 2 中提供了超参数。
WEAVER 微调。对于预训练数据集中代表性不足的复杂任务（例如倒水），世界模型的预测不够准确。为了缓解协变量偏移并改善生成结果，一个潜在的解决方案是使用任务数据集对世界模型进行微调。在本工作中，我们使用通过 VLA 收集的包含 250 条轨迹（每个任务 50 条）的小型数据集对 WEAVER 进行微调。我们使用较小的学习率对 WEAVER 进行了 16K 梯度步的微调。其他超参数与预训练阶段相似（如表 2 所述），训练在 4 块 H100 GPU 上耗时 6 小时。
微调。我们遵循 openpi 中 droid 微调配置的原始设置来微调我们的策略。我们使用开源的预训练 droid 检查点作为基础策略。归一化统计量继承自原始 DROID 检查点，并在整个微调过程中保持不变，以保持与预训练观测编码器的兼容性。所有数据集大小小于 5000 个片段的运行都使用 32 的批量大小、带有余弦衰减且无预热的峰值学习率，以及来自任务注释的语言指令。对于大小为轨迹的数据集，我们进行 10,000 个梯度步的微调，采用 10,000 步的余弦衰减和 1000 步的预热，峰值学习率为 2.5e-5。对于较小的数据集（1,000–2,000 条轨迹），我们减少训练以防止过拟合。
奖励标注。我们使用 Robometer 评估服务器 [25] 为每条轨迹标注逐帧进度奖励。对于每个片段，我们从录制的视频中提取帧，并使用 DROID 设置中的右摄像头视角，将其下采样至 1 fps。采样后的帧以及该片段的语言指令通过单次前向传递发送至 Robometer 评估服务器，该服务器返回一个逐帧进度预测，表示在帧处估计的任务完成比例，以及一个可选的逐帧成功概率。
由于视频帧以每秒1帧的速率进行子采样，生成的奖励序列比原始视频更短。我们通过线性插值将奖励重新对齐到完整视频长度：设 为原始帧数， 为推理得到的帧数，我们将采样值放置在 位置，并插值到整数网格 上。我们选择 reward_progress（插值后的进度信号）作为最终的奖励标注，因为它与实际任务结果更一致。我们将奖励进度减去1，使奖励落在 范围内，作为训练标签。
| 名称 | 数值 |
|---|---|
| 世界模型 | |
| 层数 | 32 |
| 注意力头数 | 16 |
| 嵌入维度 | 1536 |
| 注意力头维度 | 96 |
| SPRINT 概率 | 0.5 |
| 奖励模型与评论家 | |
| MLP 层数 | 2 |
| 折扣因子 | 0.995 |
| 回报 lambda | 0.95 |
| 预训练 | |
| 批次大小 | 32 |
| 批次长度 | 8 |
| 记忆帧数 (p) | 6 |
| 记忆帧步长 (m) | 5 |
| 优化器 | AdamW |
| 本体感知状态损失尺度 () | 0.1 |
| 学习率 | |
| 预热步数 | 10000 |
| EMA 衰减率 | 0.9999 |
| 训练步数 | 1000000 |
| 微调 | |
| 批次大小 | 32 |
| 学习率 | |
| 预热步数 | 200 |
| EMA 衰减率 | 0.9999 |
| 训练步数 | 16000 |
A2.3 推理
推理噪声调度。我们评估了几种确定性调度，用于将离散推理索引映射到噪声水平 ，其中 是去噪步数。我们描述了所比较的不同噪声调度——线性、sigmoid、幂函数和余弦：
其中 是逻辑 sigmoid 函数， 控制陡峭程度。对于 sigmoid 调度，我们将端点归一化为 和 。线性调度均匀分配步数，余弦和幂函数调度在低噪声区域分配更多预算，而 sigmoid 调度则将更新集中在轨迹中间部分。
整流流。为了进一步减少推理时间以及下游任务（如测试时规划）所需的NFE，我们使用ReFlow [28]对WEAVER-FT模型进行后训练，并将其命名为WEAVER-ReFlow。教师模型和学生模型均使用WEAVER-FT模型进行初始化，其中教师模型被冻结。在每个训练迭代中，我们对噪声进行采样，并使用教师模型预测未来的潜在表示。学生模型以预测的潜在表示作为目标，通过均方误差损失函数进行更新。整流流后训练在2K梯度步长上执行（耗时6小时），学习率为。
附录A3 额外世界模型评估结果
我们提供了额外的结果来评估WEAVER在连贯生成、KV缓存对推理时间的影响、噪声调度方案的优势、在任务数据上的微调以及使用整流流进行后训练等方面的表现。
A3.1 世界模型评估
对于验证数据集中的一条轨迹，我们从第20步开始生成展开序列，并使用接下来10秒的生成结果计算指标。我们使用前20帧来初始化WEAVER和Ctrl-World的记忆和历史。我们报告了基于真实视频计算得到的LPIPS [53]、FID [20]和FVD [43]。为了获得LPIPS，我们利用了torchmetrics（https://github.com/Lightning-AI/torchmetrics）中的功能，该功能使用从vgg层获得的逐帧特征。为了计算FID，我们使用了pytorch-fid（https://github.com/mseitzer/pytorch-fid）仓库中提供的实现。我们对FVD的计算使用了Style-GAN-V（https://github.com/universome/stylegan-v）仓库。在此，我们从每条轨迹中以8为步长，子采样出多个长度为16帧的子轨迹。
| 外部视角 | 腕部视角 | |||||||
| 方法 | NFE | LPIPS | FID | FVD | LPIPS | FID | FVD | 时间（秒） |
| \rowcolorpastellavender DROID | ||||||||
| Ctrl-World | 8 | 0.169 | 31.63 | 116.14 | 0.407 | 52.40 | 347.69 | 8.14 |
| 16 | 0.165 | 26.09 | 78.73 | 0.392 | 33.83 | 195.37 | 14.65 | |
| 32 | 0.168 | 23.63 | 63.55 | 0.389 | 27.14 | 114.87 | 27.67 | |
| 50 | 0.168 | 22.44 | 55.05 | 0.388 | 25.32 | 91.77 | 42.33 | |
| WEAVER | 8 | \cellcolorlightblue0.117 | 10.59 | 28.97 | \cellcolorlightblue0.372 | 24.25 | 104.53 | \cellcolorlightblue2.53 |
| 16 | \cellcolorlightblue0.117 | 10.20 | 27.83 | \cellcolorlightblue0.371 | 21.50 | 90.72 | 4.78 | |
| 32 | \cellcolorlightblue0.120 | \cellcolorlightblue 9.67 | \cellcolorlightblue25.94 | \cellcolorlightblue0.378 | \cellcolorlightblue17.53 | \cellcolorlightblue63.36 | 9.22 | |
| 50 | 0.122 | \cellcolorlightblue9.51 | \cellcolorlightblue26.54 | \cellcolorlightblue0.378 | \cellcolorlightblue16.75 | 66.89 | 14.25 | |
| \rowcolorpastelmint 新数据集 | ||||||||
| Ctrl-World | 8 | 0.193 | 48.90 | 226.29 | 0.374 | 51.26 | 434.84 | 8.14 |
| 16 | 0.182 | 36.16 | 139.54 | 0.366 | 38.76 | 277.13 | 14.65 | |
| 32 | 0.183 | 32.18 | 105.38 | 0.365 | 33.73 | 173.15 | 27.67 | |
| 50 | 0.184 | 31.44 | 91.48 | 0.367 | 33.47 | 145.86 | 42.33 | |
| WEAVER | 8 | \cellcolorlightblue0.154 | \cellcolorlightblue23.89 | 89.55 | \cellcolorlightblue0.364 | 31.70 | 193.55 | \cellcolorlightblue2.53 |
| 16 | \cellcolorlightblue 0.155 | \cellcolorlightblue23.95 | 88.27 | \cellcolorlightblue0.364 | 30.77 | 184.62 | 4.78 | |
| 32 | \cellcolorlightblue0.157 | \cellcolorlightblue23.45 | 92.36 | \cellcolorlightblue0.365 | 28.24 | \cellcolorlightblue148.85 | 9.22 | |
| 50 | \cellcolorlightblue0.159 | \cellcolorlightblue23.48 | \cellcolorlightblue87.03 | \cellcolorlightblue0.371 | \cellcolorlightblue27.37 | \cellcolorlightblue145.04 | 14.25 | |
A3.2 定量结果
表 A3.1 报告了 WEAVER 与 Ctrl-World 在 DROID 和 OOD 数据集上多项指标的对比。我们观察到，Ctrl-World 的性能随 NFE 降低而恶化，而 WEAVER 在 NFE 降低时性能仅略有下降。此外，在相似的 NFE 值下，我们的方法生成 rollout 的速度比 Ctrl-World 更快。在图 11 中，我们展示了 FID 与推理时间的对比，并观察到 NFE 低至 8 的 WEAVER 优于 NFE 高达 50 的 Ctrl-World。我们还在图 14 和图 15 中展示了不同世界模型在不同 NFE 下的定性结果。
| / NFE | 8 | 16 | 32 | 50 |
|---|---|---|---|---|
| 无 KV cache | 3.30 | 6.54 | 12.98 | 20.21 |
| 有 KV cache | 2.53 | 4.78 | 9.22 | 14.25 |
A3.3 使用 KV Cache 减少推理时间
在迭代去噪过程中，记忆帧和历史帧的潜变量以恒定的噪声水平传递。由于该噪声水平在此过程中保持不变，我们在第一步去噪时计算了记忆和历史帧潜变量的键值向量缓存。在表 4 中，我们报告了 KV Cache 可将推理时间最多降低 30%。
| 外部 | 手腕 | ||||||
| 方法 | 调度策略 | LPIPS | FID | FVD | LPIPS | FID | FVD |
| \rowcolorpastellavender DROID | |||||||
| Ctrl-World | 线性 | 0.165 | 26.09 | 78.73 | 0.392 | 33.83 | 195.37 |
| WEAVER | 线性 | \cellcolorlightblue0.117 | 11.32 | \cellcolorlightblue26.38 | \cellcolorlightblue0.375 | 24.43 | 98.82 |
| sigmoid | \cellcolorlightblue0.117 | \cellcolorlightblue10.88 | 29.76 | \cellcolorlightblue0.375 | 22.69 | 104.89 | |
| power | \cellcolorlightblue0.117 | \cellcolorlightblue10.57 | 27.93 | \cellcolorlightblue0.369 | \cellcolorlightblue21.17 | \cellcolorlightblue91.51 | |
| cosine | \cellcolorlightblue0.117 | \cellcolorlightblue10.20 | 27.83 | \cellcolorlightblue0.371 | \cellcolorlightblue21.50 | \cellcolorlightblue90.72 | |
| \rowcolorpastelmint 新数据集 | |||||||
| Ctrl-World | 线性 | 0.182 | 36.16 | 139.54 | \cellcolorlightblue0.366 | 38.76 | 277.13 |
| WEAVER | 线性 | \cellcolorlightblue0.157 | 25.37 | 96.16 | \cellcolorlightblue0.367 | 33.24 | 217.65 |
| sigmoid | \cellcolorlightblue0.156 | 24.83 | 93.15 | \cellcolorlightblue0.367 | 32.48 | 216.89 | |
| power | \cellcolorlightblue0.155 | \cellcolorlightblue23.82 | \cellcolorlightblue84.91 | \cellcolorlightblue0.363 | \cellcolorlightblue31.60 | \cellcolorlightblue185.30 | |
| cosine | \cellcolorlightblue0.155 | \cellcolorlightblue23.95 | 88.27 | \cellcolorlightblue0.364 | \cellcolorlightblue30.77 | \cellcolorlightblue184.62 | |
A3.4 推理过程中的噪声调度策略
表 A3.3 比较了不同的噪声调度方案，我们观察到幂噪声调度和余弦噪声调度的表现均优于 Sigmoid 噪声调度和线性噪声调度。由于世界模型需要以更高保真度进行生成，因此在低噪声区域分配更多带宽的噪声调度有助于生成精细的细节。
| 外部 | 手腕 | ||||||
| 方法 | NFE | LPIPS | FID | FVD | LPIPS | FID | FVD |
| \rowcolorpastelmint 任务数据（分布外） | |||||||
| Ctrl-World | 50 | 0.184 | 31.44 | 91.48 | 0.367 | 33.47 | 145.86 |
| Ctrl-World-FT | 16 | 0.140 | 29.89 | 81.16 | 0.295 | 34.77 | 283.00 |
| 50 | 0.142 | 25.96 | 58.27 | 0.292 | 25.80 | 134.69 | |
| WEAVER | 50 | 0.159 | 23.48 | 87.03 | 0.371 | 27.37 | 145.04 |
| WEAVER-FT | 4 | 0.116 | 16.62 | 50.68 | 0.304 | 33.32 | 219.15 |
| 16 | 0.118 | 14.24 | 40.50 | 0.303 | 23.39 | 146.35 | |
| 50 | 0.121 | 13.69 | 40.09 | 0.308 | 18.73 | 99.81 | |
| WEAVER-REFLOW | 4 | 0.123 | 14.95 | 44.30 | 0.312 | 23.56 | 138.67 |
A3.5 微调
在表 A3.4 中，我们观察到微调显著提升了性能，并且当 NFE=16 时，其表现优于 NFE=50 的 WEAVER。为了与 Ctrl-World 进行公平比较，我们在其上对基线进行了 20K 梯度步的微调，发现微调后的 Ctrl-World（称为 Ctrl-World-FT）表现优于预训练模型。然而，WEAVER-FT 在所有指标上均优于 Ctrl-World-FT，并且在低 NFE=16 时性能仍然更高。这进一步表明微调无助于减少 Ctrl-World 的推理时间。我们还在图 16、图 17 和图 18 中提供了由 Ctrl-World、WEAVER、WEAVER-FT 生成的展开序列的定性结果。
A3.6 使用整流流进行后训练
在表 A3.4 中，我们展示了 WEAVER-ReFlow 在低推理预算下的结果，并观察到它缩小了与使用大 NFE=16 评估的 WEAVER-FT 之间的性能差距。这使得它适用于第 3.2 节中观察到的测试时引导。
| 组件 | 符号 | 批量大小 | 视界 | 运行时间（秒） |
| 策略采样 | – | |||
| 动力学模型 | 4 | 9 | ||
| (WEAVER) | 4 | 12 | ||
| 4 | 15 | |||
| 1 | 15 | |||
| 动力学模型 | 4 | 15 | ||
| (Ctrl-World) | 1 | 15 | ||
| 奖励推理 | 4 | – | ||
| 评论家推理 | 4 | – |
附录 A4 额外下游应用结果
A4.1 策略评估结果
我们在图 12 中提供了完整的策略评估展开结果。Ctrl-World 和 WEAVER 都难以准确预测策略性能，尤其是在涉及动态操控（如倒豆子）和可变形物体操控（如袋子和毛巾操控）等具有挑战性的任务上。对于 PnP Bag 任务，抓取袋子尤其困难，因为世界模型必须从两个相机视角准确推断夹爪深度，同时还要对夹爪与可变形物体之间的接触动力学进行建模。随着预测时域的增加，这些挑战变得更加突出。相比之下，如图 19 和图 20 所示，WEAVER-FT 通过微调显著提升了评估准确性。未来的工作可以通过设计更好的记忆和历史表示来进一步改进长时域预测，使模型能够更好地推理遮挡和可变形物体的动力学。
除了皮尔逊相关系数和 MMRV，我们还报告了 RMSE 和斯皮尔曼秩相关系数 [48]。在这些指标上，我们观察到一致的趋势：WEAVER-FT 实现了最强的相关性和最低的预测误差。此外，在分布外任务数据集上的零样本策略评估中，WEAVER 优于 Ctrl-World。完整的定量结果如表 8 所示。
| 方法 | RMSE | 斯皮尔曼 | 皮尔逊 | MMRV |
|---|---|---|---|---|
| CtrlWorld | 0.410 | 0.523 | 0.552 | 0.215 |
| 我们的方法 | 0.359 | 0.594 | 0.563 | 0.155 |
| 我们的方法-FT | 0.188 | 0.870 | 0.863 | 0.035 |
A4.2 策略改进结果
我们在图13中提供了策略改进的额外定性结果。这些示例表明，基础策略常常存在抓取和放置不精确的问题，以及在动态操作过程中调整不足的情况。我们还观察到，基础策略倾向于产生更大的单步动作，导致机器人控制不稳定。相比之下，微调后的策略大幅减少了这些大幅动作，并使动作分布更加集中，从而实现了更平滑、更稳定的执行。
我们还注意到，RoboMeter 奖励标签并非完美无缺。对于 PnP Marker 任务，我们观察到奖励模型有时无法区分精细的放置精度，这可能会给预测的奖励引入噪声。未来的工作可以通过收集更多样化的失败数据来训练更通用、更精确的奖励模型，从而改进奖励监督。为了减轻噪声奖励标签的影响，我们将优势阈值设置为 ，这有助于防止低质量片段被选中用于微调，从而避免潜在的性能下降。如图4所示，我们的过滤流程能够从候选动作中选出最佳样本。
A4.3 测试时规划结果
推理时延迟。表7报告了在 A6000 Ada GPU 上进行测试时规划的推理时时间分解。总体而言，运行时间主要由动力学模型想象主导，而奖励和评论家推理的时间可以忽略不计，每次均少于 秒。对于 WEAVER，动力学运行时间随想象步长适度增加：从步长 时的 秒，到步长 时的 秒，再到批大小为 时步长 的 秒。包括策略采样、奖励推理和评论家推理在内，步长分别为 、 和 时的完整规划延迟大约为 秒、 秒和 秒。
与 Ctrl-World 相比，WEAVER 显著降低了想象过程中的延迟。在给定的预测步长和批次大小下，WEAVER 进行动力学预测需要 s，而 Ctrl-World 需要 s，实现了 倍的加速。在批次大小为 时也呈现相同趋势，WEAVER 耗时 s，Ctrl-World 耗时 s，实现了 倍的加速。这些结果表明，WEAVER 能够实现延迟更低的测试时规划，使得在策略执行过程中重复进行世界模型想象变得切实可行。
附录 A5 局限性
尽管 WEAVER 展示了大规模世界模型在策略评估、策略改进和测试时规划方面的潜力，但仍存在若干局限性。
A5.1 部分可观测性
我们的世界模型主要依赖视觉观测，而视觉观测只能提供对底层物理状态的部分访问。在操作过程中，与任务相关的信息，如物体接触、抓取稳定性、施加的力或被遮挡的物体几何形状，可能从所有可用的摄像头视角中隐藏。这一局限性在腕部摄像头观测中尤为明显，因为其视角会持续变化；在杂乱场景中，物体可能离开视野或被夹爪遮挡。尽管记忆和多视角条件化能缓解这一问题，但当缺失状态无法仅从图像历史中推断时，纯视觉预测仍可能失败。引入额外的感知模态，如触觉反馈、力-力矩传感或深度信息，可能有助于改善遮挡情况下的状态估计和长程预测。
A5.2 复杂可变形与动态交互
对于可变形物体的操作以及动态操作，对于学习型世界模型而言仍然具有挑战性。毛巾、袋子、颗粒材料等物体表现出高维度、依赖历史状态的动力学特性，这些特性很难从有限的机器人数据中捕捉到。在预测接触点、抓取位置或物体构型时出现的微小误差，会随时间累积，导致定性上错误的推演结果。这在倾倒等任务中尤为明显，因为颗粒材料的运动对杯子的姿态、速度以及与容器的接触高度敏感。未来的工作可以通过引入物理先验、混合神经-物理模型，或专门针对可变形和颗粒动力学设计的神经模拟器，来提高预测的保真度。
A5.3 测试时有限的规划视野
尽管我们的推理加速策略使得在测试时使用大型生成式世界模型进行规划成为可能，但延迟仍然将在线规划限制在单个动作块内。因此，规划器可以改进近期的动作选择，但尚无法执行长视野的前瞻。这限制了其推理延迟后果或多阶段恢复行为的能力。在采样效率、模型蒸馏、价值估计或分层规划方面的进一步改进，将能够在保持实时控制的同时，实现更长视野的在线推理。
A5.4 数据覆盖范围与具身多样性
我们的世界模型主要在 DROID 数据集上进行预训练，该数据集提供了大规模的机器人交互数据，但仍受限于特定的机器人具身形态和数据采集设置。这可能会限制模型向差异显著的机器人、相机配置和末端执行器进行泛化。此外，我们评估中的某些任务动力学，例如颗粒倾倒，在预训练数据中的代表性不足。将世界模型的训练扩展到更多样化的数据源，包括跨具身形态的机器人数据集、仿真数据以及人类视频，可能会提高模型的鲁棒性，并拓宽能够被准确想象的行为范围。
A5.5 有噪声的奖励监督
我们的潜在奖励和评论家头使用现成奖励模型的标签进行训练。虽然这能实现高效的潜在空间评估，但由此产生的监督信号可能带有噪声或不完整，尤其是在处理细微故障模式时。例如，奖励模型可能无法区分视觉上相似但语义不同的结果，或者对接触、放置或任务完成中的微小错误不敏感。这种噪声会影响策略评估和下游策略改进。一个基于大规模机器人成功与失败数据训练的、更可靠的奖励模型（可能带有校准后的不确定性）将有望提高想象展开评估的可靠性。
总体而言，这些局限性表明，未来的进展不仅需要更大、更快的世界模型，还需要更丰富的感知、更广泛的数据覆盖、更强的物理归纳偏置以及更准确的奖励监督。
Equal Contribution. Correspondence to Arnav <arnav-kumar.jain@mila.quebec> and Yilin <yilinwu@andrew.cmu.edu>.
Abstract
The potential impacts of world models (WMs, i.e., learned simulators) on robotics are far-reaching—policy evaluation, policy improvement, and test-time planning—all with limited real-world interaction. To unlock these downstream capabilities, a WM needs to jointly satisfy three desiderata: (i) fidelity (i.e., producing simulated trajectories that correlate with reality), (ii) consistency (i.e., producing simulated trajectories that are coherent over long horizons), and (iii) efficiency (i.e., producing simulated trajectories quickly). We propose WEAVER (World Estimation Across Views for Embodied Reasoning): a WM architecture that simultaneously achieves all three desiderata, providing state-of-the-art results on robotic manipulation tasks. WEAVER is a multi-view WM trained to predict future latents and reward values via a flow-matching loss. We distill the key design decisions across model architecture, memory, and prediction objectives required to unlock the kinds of long-horizon dynamic manipulation tasks that have confounded prior world modeling approaches. We apply WEAVER in robotic hardware, demonstrating its effectiveness at policy evaluation ( correlation with real-world success rate), policy improvement (real-world success rate improvement of on top of the robot foundation model), and test-time planning (real-world success rate improvement of with a speedup over prior WMs). WEAVER also demonstrates better performance than prior WMs when evaluated on out-of-distribution scenarios. Code, models, and videos at: https://arnavkj1995.github.io/WEAVER/.
1 Introduction
World models (WMs, [13]), or learned simulators, have attracted intense interest from both academia [12, 54, 51, 33] and industry [6, 36]. This is because of the tremendous promise of WMs for robotics: the ability to both evaluate and improve policies without costly and often unsafe real-world interaction. Furthermore, WMs unlock test-time scaling when incorporated into planning algorithms.
To simultaneously deliver on the three promises of evaluation, improvement, and planning, a robot WM must jointly satisfy three core desiderata. The first is (i) fidelity: producing physically accurate predictions that correlate with real-world outcomes. The second is (ii) consistency: producing predictions that remain coherent over long prediction horizons. The third is (iii) efficiency: producing predictions quickly. For example, policy evaluation and improvement require high-fidelity predictions (for handling arbitrary, visuomotor robot policies) as well as consistency (to handle multi-stage tasks). Relatedly, planning requires fast inference for dealing with the real-time requirements of robots.
Despite rapid progress, no existing robot WM satisfies all three desiderata in tandem. For example, video generation models [29] produce high fidelity generations at the cost of low efficiency. Similarly, JEPA-style WMs [2] have latent states that may not be decodable into the images required to evaluate arbitrary visuomotor robot policies. And while Dreamer-v4 [16] appears promising, learning an encoder from scratch rather than using a pretrained model can harm out-of-distribution robustness.
When we focus on robotic manipulation, the world modeling problem becomes even more complex, as we must handle multiple views of the scene, infer occluded objects from history, and ensure relatively high fidelity predicted world states rather than just visual aesthetics. Handling these complexities often comes at the cost of efficiency, with state-of-the-art WMs for manipulation like Ctrl-World [12] operating at far slower speeds than the real world, precluding their use in test-time planning and making policy improvement computationally challenging.
In response, we introduce WEAVER (World Estimation Across Views for Embodied Reasoning): a WM architecture that achieves (i) high fidelity, (ii) long-horizon consistency, and (iii) efficient generation, unlocking state-of-the-art performance across policy evaluation, improvement, and test-time planning on challenging robotic manipulation tasks. To achieve this trifecta of capabilities, WEAVER fuses together key design decisions from prior world modeling approaches. From the video generation community, we adopt diffusion forcing [7] and flow matching [27] (for long-horizon generation at fast inference speeds) and the use of a pretrained encoder [35] (for out-of-distribution robustness). From latent world models [16, 37, 12], we adopt the use of a reward prediction head to facilitate efficient evaluation without the need for an external judge model like a VLM. From JEPA [3], we adopt future latent prediction (rather than image reconstruction) as our primary training objective. Lastly, to handle the particular complexities of robot manipulation, we adopt the multi-view generation and memory architecture of Ctrl-World [12].
Put together, we end up with a gestalt whole: a WM for robotic manipulation that can be used flexibly across evaluation, improvement, and planning. On a suite of five manipulation tasks (from pick and place to deformable object manipulation) performed on real hardware, WEAVER demonstrates strong correlation () with real-world success rate when used for evaluation, improves the real-world success rate of the [21] robot foundation model by without any real-world interaction, and unlocks test-time planning faster than Ctrl-World [12].
2 Related Work
Robot World Models. While world models have been explored across autonomous driving [36, 45], video games [18], and code generation [8], we focus on their application to robotics [49, 12, 37, 42, 3] – more specifically visual manipulation. While improvements in video generation [47, 4] have lead to high (i) fidelity WMs [29, 12, 37, 33, 12, 10, 36], these WMs are often not (iii) efficient enough to use for test-time planning. However, incorporating key ingredients from the broader vision community, like flow matching [27], diffusion forcing [7] allows us to improve the (iii) efficiency of WEAVER. Furthermore, the use of pretrained video generation model encoders [35] enhances WEAVER’s robustness to out-of-distribution visual inputs, while the use of pretrained decoders allows us to evaluate arbitrary visuomotor robot policies unlike JEPA-style models [3]. Finally, we adopt the latent reward and value heads of Dreamer-v4 [16] to enable (iii) efficient evaluation and planning without the need to pass decoded images to an external and often slow VLM judge like in [11].
Prior WMs [14, 15, 17, 49, 50, 23, 22] struggle to maintain temporal (ii) consistency across long horizons. In response, we adopt the use of multi-view prediction, history, and memory from [12, 36] to ensure generations remain coherent even when gripper-object interactions are under occlusions. This is in contrast to earlier WMs like WorldGym [33], DreamerV4 [16] and DreamDojo [10].
Perhaps the most similar approaches to our own are Ctrl-World [12] and Dreamer-v4 [16]. By using techniques from the video generation community [7, 27] for more (iii) efficient inference, we are able to produce higher (i) fidelity generations that are more temporally (ii) coherent in less time, Pareto dominating Ctrl-World [12]. By using a pretrained encoder [35] instead of learning one from scratch as in Dreamer-v4 [16], we likely inherit better robustness to out-of-distribution visual inputs.
Uses of World Models in Robotics. World models promise “downstream” advances in robotic policy evaluation, improvement, and test-time planning. Prior work has shown that sufficiently faithful world models can enable scalable policy evaluation [42, 51, 46], while early results suggest that synthetic trajectories may also improve policies [11, 46], though the extent to which this is true remains an open question. More recently, world models have been explored for test-time planning [32, 50], where the central challenge is generating accurately quickly for online optimization. WEAVER is designed with each of these downstream applications in mind for robotic manipulation.
3 WEAVER: World Estimation Across Views for Embodied Reasoning
We now describe the key ingredients in WEAVER: a robot world model designed to support policy evaluation, policy improvement, and test-time planning. These downstream applications of the WM on manipulation tasks imply three key desiderata upstream: (i) fidelity across multiple views during physical interaction, (ii) consistent predictions across long-horizon interactions that can introduce occlusions, and (iii) efficient enough generation for use in a real-time planning algorithm.
To jointly satisfy these three desiderata, WEAVER fuses together a variety of ingredients. We first describe the key WM design decisions and training objective (Sec. 3.1), followed by inference acceleration (Sec. 3.2) and latent-space value estimation (Sec. 3.3). We then show how, when put together, these components enable evaluation, improvement, and planning (Sec. 3.4).
Setup: Robot & Policy. We consider long-horizon robotic manipulation tasks specified by a natural language instruction . Let the robot’s proprioceptive state (e.g., joint angles) be denoted by . The robot also has RGB views of the scene (e.g., from wrist and third person cameras); let this set of multi-view images be . At timestep , the robot observes both the multiview images and proprioceptive state: . Let the robot’s action be denoted by (e.g., joint velocities). Given any , the robot’s base policy generates , an -step future action chunk (i.e., ), which is then executed in the WM / environment.
World Model Architecture. Our WM maps an observation into a latent state via a pretrained encoder . A key design choice is conditioning our world model on both a memory of every th prior latent, , as well as a -step history of the most recent latents, . Given memory, history, and an -step action plan , the WM predicts future latents:
| (1) |
where is the -step future. We also train a reward model that scores the predicted latent’s alignment with the language instruction: where . To enable iterative calls to the visuomotor policy, we use a pretrained decoder to obtain future observations (camera views and proprioceptive state), , where are the -step future observations. The final prediction, , is fed back to the policy to generate the next action chunk.
3.1 Key Design Decisions for High Fidelity, Temporally Consistent World Model Generation
Multi-View Camera Prediction. Although using multiple views (e.g., wrist and external cameras) is common practice when designing visuomotor robot policies for manipulation to handle partial observability and capture finer-grained object–gripper interactions [21], many WMs only predict a single view [33, 10]. Following [12, 24, 44], WEAVER predicts both external and wrist-camera observations. The increased information provided by multiple views helps with (ii) consistency by helping handle occlusions during manipulation. Each view is encoded into patch tokens using the pretrained Stable Diffusion 3 VAE encoder [9]. We project the proprioceptive state to the same token dimension and obtain by concatenating patch tokens and the proprioceptive token.
Proprioceptive State Prediction. In addition to future visual latents, WEAVER also predicts future proprioceptive states. We find that explicitly predicting the robot’s configuration (rather than just visual observations like Ctrl-World [12]) is critical to handle contact-rich manipulation of deformable objects, where knowing the precise position of the arm and width of the gripper is often required.
Sparse Memory and Short-Term History. Temporal (ii) consistency across WM generations requires the WM to understand both what changes and what stays the same across an interaction. This is particularly challenging in manipulation, where occlusions and wrist camera viewpoint changes can cause objects and parts of the background scene to leave and enter the robot’s FOV. In response, WEAVER builds upon [12] and conditions on two sets of observations when generating futures: a long-term, sparse memory, and a short-term history. In particular, memory includes every th encoded observation to help capture longer-term context, while history includes the last two frames to capture the shorter-term consequences of actions.
Latent Dynamics Model. The latent dynamics model predicts future latent states conditioned on memory, history, and a candidate action plan. To balance (i) fidelity with (iii) efficiency, WEAVER adopts an efficient 2D transformer architecture following [16, 33], with dynamics blocks composed of spatial attention and causal temporal attention. At each prediction step, the model conditions on latent tokens, action tokens, and flow timestep embeddings to autoregressively generate an -step chunk. For stable training, each block uses RMSNorm [52], RoPE [39], QKNorm [19], and SwiGLU feed-forward layers [38] (see A2 for more details).
Training Objective. Similar to [16], we train our latent dynamics model with a flow-matching loss [27] to predict future latents. Let denote the ground-truth next latents and let denote a Gaussian noise vector of the same dimension. Next, we define , with . Then, we train to predict “velocity” by minimizing mean squared error: . To improve long-horizon (ii) consistency, we adopt Diffusion Forcing [7], which trains the latent dynamics model with independently sampled noise levels across future timesteps. We also use SPRINT blocks [30], which aggressively drop patch tokens in the latents to improve (iii) efficiency.
3.2 Accelerating World Model Inference Speed
For diffusion transformer-based WMs [12, 16, 33] like WEAVER, latency is a product of both (a) the forward pass through the model and (b) iterative denoising. Thus, (iii) efficient generation requires tackling both of these concerns in tandem. We reduce cost (a) via the use of KV caching to memory and history tokens across denoising steps. We reduce cost (b) by adjusting the denoising process. In particular, building on diffusion forcing [7], we use a progressive noise schedule. Rather than using a linear schedule like in [12, 16], WEAVER adopts a cosine schedule for higher (i) fidelity generation.
To further increase (iii) efficiency to the level required for test-time planning, we post-train WEAVER with a rectified flow objective [28] to enable high-quality generation within a few forward passes. In particular, we first generate a high-quality latent trajectory using the denoising process, before using it as a target for secondary distillation step. See Appendix A2.3 for more implementation details.
3.3 Accurate and Efficient Value Estimation from the World Model
Reward Model. To enable (iii) efficient scoring of a proposed action chunk without needing to (a) decode a latent into an image and (b) feed it to an external VLM judge model [11, 33], we distill the scores produced by an off-the-shelf reward model into a lightweight reward head that operates directly on latent states and language instruction . The reward head aggregates latent tokens with AdaPool [5], followed by MLP layers. We train with a simple mean squared error objective.
Critic. To support truncated-horizon rollouts with the WM, WEAVER learns a critic network that estimates the value beyond the imagined horizon. The critic shares the same latent-space design as the reward model and is trained with an MSE objective to predict bootstrapped -returns [40]. Given latent rewards from , the target is defined recursively as , . The critic is then trained by minimizing .
3.4 Downstream WM Applications: Evaluation, Improvement, Planning
By satisfying the desiderata of (i) fidelity, (ii) consistency, and (iii) efficiency simultaneously, WEAVER can support the downstream capabilities of evaluation, improvement, and planning.
Policy Evaluation. For policy evaluation, we take recorded action trajectories from real-world rollouts and execute them open-loop inside WEAVER, recording predicted reward values along the way. We focus on long-horizon tasks that sometimes require 40+ iterative evaluations of WEAVER’s latent dynamics model, underscoring the importance of temporal (ii) consistency and (iii) efficiency.
Policy Improvement. For policy improvement, we sample a -step action chunk from the policy and forward simulate inside the WM times for a total of timesteps, leveraging WEAVER’s (i) fidelity and (ii) consistency. After doing this times from the same initial observation , we collect batch of rollouts . We then compute a Monte-Carlo estimate of the -step advantage along each rollout: If the highest-scoring rollout in the batch (i.e., ) has an advantage value above some small, positive threshold (i.e., ), we distill it into the base policy. This advantage-based filtering prevents the policy from being updated at states where all -step sampled plans are predicted to be worse than the current expected behavior of the policy [23, 1].
Test-time Planning. We adopt a single-chunk, best-of- [26] approach to test-time scaling that doesn’t involve iteratively calling the latent dynamics model. In particular, given the current observation and instruction, we sample candidate action chunks from the policy, imagine their outcomes with the world model, and execute the one with the highest advantage estimated with latent reward and critic heads. WEAVER’s (iii) efficiency (both in terms of the speed of the latent dynamics model and ability to evaluate a candidate action sequence without needing to call an external VLM judge via the the use of the reward head) are critical to unlocking this test-time scaling capability.
4 Experimental Setup
Base Policy & Hardware. Our base policy is [21], a state-of-the-art vision-language-action (VLA) policy trained on the DROID dataset [41]. We follow the DROID hardware setup and use a single Franka Emika Panda manipulator, two external Zed 2i cameras mounted on the left and right sides of the workspace, and a wrist-mounted Zed Mini camera (see Figure 10 in Appendix). The VLA policy and our WEAVER world model use only the right camera view and the wrist camera111We setup all three cameras because our main world model baseline [12] uses all three views..
Datasets & Tasks. To align the world model with the data distribution of the base policy, we first pre-train the WEAVER world model on the DROID dataset and then fine-tune it on our real-world setup. We collect data to fine-tune the world model by running for five real-world manipulation tasks, with 50 rollouts per task. We also collect an additional 20 rollouts per task as evaluation data . We select tasks such that the base policy achieves at least success rate while spanning a range of capabilities from rigid object pick-and-place to deformable object manipulation and dynamic manipulation. Specifically, our tasks are: Stack Bowls (stack one bowl on another); PnP Bag (place a deformable chip bag onto a plate); PnP Marker (reorient a marker and insert it into a cup); PnP Towel (place a soft towel into a basket); and Pour Beans (pour a cup full of coffee beans into a bowl). Details on each task can be found in Appendix A1.1.
World Model Training. WEAVER is a 928M parameter model. We pretrain on the DROID dataset [41] for 1M steps with a batch size of 32 and learning rate of on GPUs for 10 days. For training the reward model and critic on top of WEAVER’s latents, we annotate the DROID dataset with progress-rewards obtained from Robometer [25] (reduced by 1 to get negative rewards). During world model finetuning, the model is updated with a lower learning rate of for 16k steps on our collected task data. The resulting model is used for policy evaluation, policy finetuning, and test-time planning. Like prior work [12], we downsample the steps by 3 to use frequency of 5Hz for world model imagination. We represent actions as the joint position difference between two timesteps to match the action space of the policy. We learn an additional joint-velocity-to-position action adapter to convert between the action spaces for data generation and test-time planning (see A1.2).
5 Results
We first study the performance of the WEAVER world model in isolation (Sec. 5.1) and then in the downstream use-cases of policy evaluation, improvement, and test-time planning (Sec. 5.2).
5.1 WEAVER Pareto-Dominates leading Manipulation World Models
We start by comparing the performance of WEAVER pre-trained only on the DROID dataset to leading multi-view manipulation world model. Ctrl-World [12] is a 1.5B-parameter diffusion model trained on the DROID dataset and is initialized from a pretrained SVD checkpoint [4].
Setup & Metrics. We evaluate both models on a validation split of the DROID dataset (256 trajectories) and an out-of-distribution dataset collected using VLA (100 trajectories). For each trajectory, the models are rolled out autoregressively to generate 10s long sequences where each generation predicts the outcome of 15-step action chunks (1s) jointly. Following prior evaluations [12], we measure the visual fidelity of the decoded generations using FID [20], and FVD [43] computed with the ground-truth videos. More metrics are detailed in the Appendix A3.1.
| Exterior | Wrist | Time | ||||
| Method | NFE | FID | FVD | FID | FVD | (s) |
| \rowcolorpastellavender DROID (val) | ||||||
| Ctrl-World | 16 | 26.09 | 78.73 | 33.83 | 195.37 | 14.65 |
| 50 | 22.44 | 55.05 | 25.32 | 91.77 | 42.33 | |
| WEAVER | 16 | \cellcolorlightblue10.20 | 27.83 | 21.50 | 90.72 | \cellcolorlightblue4.78 |
| 50 | \cellcolorlightblue9.51 | \cellcolorlightblue26.54 | \cellcolorlightblue16.75 | \cellcolorlightblue66.89 | 14.25 | |
| \rowcolorpastelmint Task data (OOD) | ||||||
| Ctrl-World | 16 | 36.16 | 139.54 | 38.76 | 277.13 | 14.65 |
| 50 | 31.44 | 91.48 | 33.47 | 145.86 | 42.33 | |
| WEAVER | 16 | \cellcolorlightblue23.95 | \cellcolorlightblue88.27 | 30.77 | 184.62 | \cellcolorlightblue4.78 |
| 50 | \cellcolorlightblue23.48 | \cellcolorlightblue87.03 | \cellcolorlightblue27.37 | \cellcolorlightblue145.04 | 14.25 | |
Results: Perceptually High Fidelity Generations. Table 1 compares FID and FVD results for WEAVER and Ctrl-World on different evaluation datasets. WEAVER outperforms the Ctrl-World while having lower inference time. As we decrease the number of function evaluations (NFE) to decrease the latency, we find that the quality of Ctrl-World decreases more significantly than WEAVER; both models incur the highest error when predicting wrist camera viewpoints. We provide additional results to compare NFEs (Appendix A3.2), different noise schedules (Appendix A3.4) and inference speedup obtained with KVcaching (Appendix A3.3).
Results: Higher Quality at Long Horizon. We next measure how the perceptual quality of the world model’s imaginations are influenced by long-horizon predictions. For both world models, we generate rollouts with long (150-step or 10s) action sequences and measure the FID for each predicted video of the 15-step interval to estimate the generation quality with time horizon. As shown in Fig. 3, on the DROID dataset, we find that WEAVER maintains consistently lower FID compared to Ctrl-World even as inference budgets are reduced from 50 to 16 NFE. On the OOD dataset, WEAVER maintains the performance gap on exterior-view and has comparable performance on the wrist-view.
Results: WEAVER Pareto-Dominates Inference Speed vs. Quality. Next, we study how the generation quality is influenced by a fixed inference time budget as measured by NFEs and the inference time to generate 10s chunk on a single H100 GPU. In Fig. 5, we see that WEAVER significantly outperforms Ctrl-World at NFEs from 8,16,32,50 while enjoying significantly lower inference speeds (e.g., 30-50s with Ctrl-World vs. 10-30s for WEAVER). By pareto-dominating Ctrl-World, WEAVER unlocks faster evaluation and planning as we explore below in Section 5.2.
Results: Latent Reward Prediction Accuracy. Finally, we compare WEAVER’s latent reward prediction to the reward labels from RoboMeter [25], evaluated on real held-out trajectories. Fig. 4 shows the predicted reward for a rollout of the PnP Stack task; WEAVER correctly imagines key events such as grasping and stacking and the reward of the imaginations correlates with the ground-truth RoboMeter reward. In the right panel of Fig. 4, we see that the advantage computed with the predicted reward is also able to distinguish different outcomes of the action samples. This is a promising indicator that WEAVER and it’s latent reward are suitable for filtering synthetic data in Sec. 5.2.1 and test-time planning in Sec. 5.2.3.
5.2 WEAVER Enables Effective Evaluation, Improvement and Planning
Thus far, we have validated that WEAVER effectively balances (i) fidelity, (ii) long-horizon consistency, and (iii) efficient generation. Next, we turn to the downstream uses of a world model: policy evaluation (Sec. 5.2.1), policy improvement (Sec. 5.2.2), and test-time planning (Sec. 5.2.3).
5.2.1 WEAVER Enables Effective Policy Evaluation that Tightly Correlates with Reality
First, we evaluate whether WEAVER can serve as a learned simulator for offline policy evaluation, reducing the need for costly real-world rollouts.
Setup. Given an initial real observation, , and action sequence, , we autoregressively generate imagined observations and estimate policy performance from the resulting rollout. We compare three world models: Ctrl-World pretrained on DROID [12], WEAVER pretrained on DROID, and WEAVER-FT finetuned on . To test robustness of the world models across base policy quality, we evaluate each model on rollouts from both the base policy and a finetuned policy.
Metrics. Following prior work [51], we measure how well performance of generated rollouts correlates with real-world performance by comparing human-labeled binary success rates on imagined rollouts with real success rates on , averaged over 20 trials per task. We report Pearson Correlation coefficient [31] and maximum matrix ranking violation (MMRV) [42] (see Appendix A4.1).
Results. Fig. 6 shows that pretrained world models tend to underestimate policy performance, but WEAVER achieves better agreement with real rollouts than Ctrl-World, with higher Pearson correlation and lower MMRV. This setting is challenging because rollouts can last up to 40 seconds and require accurate long-horizon prediction. The pouring task is particularly difficult for pretrained models, likely because granular dynamics are underrepresented in DROID and inherently hard to model. After finetuning, WEAVER-FT substantially improves evaluation accuracy, increasing Pearson correlations to and better matching real outcomes across policies of varying performance. The qualitative example on the left of Fig. 6 further shows that WEAVER-FT predicts the PnP Towel and Pour Beans task outcomes more accurately than the baselines.
5.2.2 WEAVER Enables Effective Policy Improvement Without Real Interactions
Another desirable use of high-fidelity world models is synthetic data generation for policy improvement. We use the world model to sample and verify candidate action segments, then distill high-value imagined segments back into the policy [23, 1].
Setup. To evaluate the utility of WEAVER towards improving policies, we explore various strategies to generate data for finetuning the policy: (1) Base Policy: the original VLA trained on DROID; (2) FT w/ Real Data: we prune segments in real trajectories using advantage estimates, yielding 1,000 segments of 36-step action chunks per task; (3) FT w/ Synthetic Data: we sample multiple segments using the base policy and WEAVER, filter them based on predicted advantage values (Sec. 3.4), and retain 1,000 segments per task, and (4) FT w/ Mixed Data: combine filtered real and synthetic datasets (2000 segments per task) (more details and results are presented in App. A2.2 & A4.2).
Results. Fig. 7 shows that all finetuned policies substantially improve their success rate over the base policy. Notably, finetuning on synthetic data closely matches that on real data, with only a average performance gap. This indicates that out synthetic data is of such a high quality that it unlocks similar policy improvement to costly real world data. Combining real and synthetic data further improves performance, increasing the average success rate by over real-data finetuning alone. These results suggest that imagined rollouts from the world model provide a useful source for distillation, reducing the need for costly real-world collection and manual filtering. Fig. 7 also shows improvements on contact-rich and dynamic manipulation tasks, such as more precise marker placement and bean pouring. We further study synthetic data scaling on the Pour Beans task by varying the number of imagined segments from 1,000 to 2,000 and 5,000. Fig. 7 (right) shows that policy performance improves consistently with more synthetic data, eventually exceeding the performance obtained from real-data finetuning alone.
5.2.3 WEAVER Enables Test-Time Planning by Balancing Inference Speed and Quality
Finally, test-time search requires evaluating multiple action sequences before execution, making inference speed a key bottleneck. In contrast to planning in the the image space using reconstruction and VLM-as-a-judge [11, 37], WEAVER plans in the latent space for greater efficiency [15, 23].
Setup. We use as the base policy and sample a batch of action chunks. For each chunk, WEAVER imagines latents of future states and evaluates the advantage using the reward and critic heads. This reduces the cost of decoding predicted observations and querying external VLM judges. Following the policy-improvement setup from Sec. 5.2.2, we evaluate test-time planning on five tasks and compare against the base policy. We use parallel samples and an imagination horizon of , balancing planning quality and latency.
Results. We report task success rate and the inference-time breakdown in the test-time planning pipeline. Fig. 9 shows that advantage-based selection steers the policy toward successful behaviors. Test-time planning improves the average success rate by over the base policy, with maximum gain up to . The improvement is larger when the base policy is weaker, although it remains smaller than direct finetuning because planning is limited to a single action chunk and must operate under latency constraints. Table 7 in Appendix A4.3 shows dynamics prediction remains the main computational bottleneck. Nevertheless, WEAVER is about faster than Ctrl-World inference pipeline [12] on an RTX A6000 Ada GPU, and batched sampling scales sublinearly with the number of candidates, showing that our inference optimizations make world-model-based test-time planning practical for real-time manipulation.
6 Conclusion
We introduce WEAVER: a World Model for manipulation that achieves (i) high fidelity, (ii) is temporally coherent, and (iii) generates efficiently. Across tasks, WEAVER shows strong correlation () with real-world success rate for evaluation, improves the success rate of the policy by without any real-world interaction, and unlocks test-time planning faster than Ctrl-World [12].
Limitations. While WEAVER unlocks the potential of large-scale world models for manipulation, several limitations remain. First, visual world models observe only a partial view of the underlying state, and tactile sensing may be necessary to resolve ambiguities. Second, incorporating physics priors could improve performance on tasks involving deformable-object manipulation. Third, generation latency currently limits test-time planning to short-horizon reasoning over a single action chunk. Finally, reward supervision from RoboMeter can be noisy, motivating the development of better reward models for failure prediction. We provide further discussion in Appendix A5.
Broader Impact. This work explores large-scale world models to improve the efficiency, safety, and scalability of robotic manipulation by reducing reliance on costly real-world interaction. Imagined rollouts can support policy evaluation, improvement, and test-time planning before execution, but inaccurate or biased predictions may lead to risky decisions that are particularly important in safety-critical domains like assistive robots. Responsible deployment therefore requires careful validation, uncertainty estimation, and safeguards against exploiting errors in learned world or reward models.
Acknowledgments
We would like to thank Jesse Zhang for helpful discussions about reward models and ROBOMETER. AJ is supported by Fonds de Recherche du Quebec (FRQ) (DOI assigned: https://doi.org/10.69777/350253), Calcul Quebec, and Canada Excellence Research Chairs (CERC) program. GKS is supported by a STTR grant. The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada (https://alliancecan.ca) and Mila (https://mila.quebec). YW and AB were partially supported by the National Science Foundation (NSF) award and NSF CAREER award ]. The views expressed are those of the authors and do not necessarily reflect those of NSF.
References
- Anthony et al. [2017] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Neural Information Processing Systems (NeurIPS), 2017.
- Assran et al. [2023] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR), 2023.
- Assran et al. [2025] Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. CoRR, abs/2506.09985, 2025.
- Blattmann et al. [2023] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, abs/2311.15127, 2023.
- Brothers [2025] Greyson Brothers. Robust noise attenuation via adaptive pooling of transformer outputs. In International Conference on Learning Representations (ICLR), 2025.
- Bruce et al. [2024] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024.
- Chen et al. [2024] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In Neural Information Processing Systems (NeurIPS), 2024.
- Copet et al. [2025] Jade Copet, Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, et al. Cwm: An open-weights llm for research on code generation with world models. CoRR, abs/2510.02387, 2025.
- Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024.
- Gao et al. [2026] Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, et al. Dreamdojo: A generalist robot world model from large-scale human videos. CoRR, abs/2602.06949, 2026.
- Guo et al. [2026a] Yanjiang Guo, Tony Lee, Lucy Xiaoyang Shi, Jianyu Chen, Percy Liang, and Chelsea Finn. Vlaw: Iterative co-improvement of vision-language-action policy and world model. CoRR, abs/2602.12063, 2026a.
- Guo et al. [2026b] Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model for robot manipulation. In International Conference on Learning Representations (ICLR), 2026b.
- Ha and Schmidhuber [2018] David Ha and Jürgen Schmidhuber. World models. CoRR, abs/1803.10122, 2018.
- Hafner et al. [2020] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020.
- Hafner et al. [2021] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations (ICLR), 2021.
- Hafner et al. [2025] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. CoRR, abs/2509.24527, 2025.
- Hansen et al. [2022] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), 2022.
- He et al. [2025] Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game 2.0: An open-source real-time and streaming interactive world model. CoRR, abs/2508.13009, 2025.
- Henry et al. [2020] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
- Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Neural Information Processing Systems (NeurIPS), 2017.
- Intelligence et al. [2025] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi0.5: a vision-language-action model with open-world generalization. CoRR, abs/2504.16054, 2025.
- Jain et al. [2022] Arnav Kumar Jain, Shiva Kanth Sujit, Shruti Joshi, Vincent Michalski, Danijar Hafner, and Samira Ebrahimi Kahou. Learning robust dynamics through variational sparse gating. In Neural Information Processing Systems (NeurIPS), 2022.
- Jain et al. [2026] Arnav Kumar Jain, Vibhakar Mohta, Subin Kim, Atiksh Bhardwaj, Juntao Ren, Yunhai Feng, Sanjiban Choudhury, and Gokul Swamy. A smooth sea never made a skilled SAILOR: Robust imitation via learning to search. In Neural Information Processing Systems (NeurIPS), 2026.
- Jiang et al. [2025] Yuxin Jiang, Shengcong Chen, Siyuan Huang, Liliang Chen, Pengfei Zhou, Yue Liao, Xindong He, Chiming Liu, Hongsheng Li, Maoqing Yao, et al. Enerverse-ac: Envisioning embodied environments with action condition. CoRR, abs/2505.09723, 2025.
- Liang et al. [2026] Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons. In Robotics: Science and Systems 2026, 2026.
- Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024.
- Lipman et al. [2023] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.
- Liu et al. [2023] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023.
- Mei et al. [2026] Zhiting Mei, Tenny Yin, Ola Shorinwa, Apurva Badithela, Zhonghe Zheng, Joseph Bruno, Madison Bland, Lihan Zha, Asher Hancock, Jaime Fernández Fisac, et al. Video generation models in robotics-applications, research challenges, future directions. CoRR, abs/2601.07823, 2026.
- Park et al. [2026] Dogyun Park, Moayed Haji-Ali, Yanyu Li, Willi Menapace, Sergey Tulyakov, Hyunwoo J Kim, Aliaksandr Siarohin, and Anil Kag. Sprint: Sparse-dense residual fusion for efficient diffusion transformers. In International Conference on Learning Representations (ICLR), 2026.
- Pearson [1920] Karl Pearson. Notes on the history of correlation. Biometrika, 13(1):25–45, 1920.
- Qi et al. [2026] Han Qi, Haocheng Yin, Aris Zhu, Yilun Du, and Heng Yang. Inference-time enhancement of generative robot policies via predictive world modeling. IEEE Robotics and Automation Letters, 11(5):5534–5541, 2026.
- Quevedo et al. [2026] Julian Quevedo, Ansh Kumar Sharma, Yixiang Sun, Varad Suryavanshi, Percy Liang, and Sherry Yang. Worldgym: World model as an environment for policy evaluation. In International Conference on Learning Representations (ICLR), 2026.
- Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
- Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
- Russell et al. [2025] Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving (2025). CoRR, abs/2503.20523, 2025.
- Sharma et al. [2026] Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, and Sherry Yang. World-gymnast: Training robots with reinforcement learning in a world model. CoRR, abs/2602.02454, 2026.
- Shazeer [2020] Noam Shazeer. Glu variants improve transformer. CoRR, abs/2002.05202, 2020.
- Su et al. [2024] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
- Sutton et al. [1998] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
- Team [2024] DROID Team. Droid: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems, 2024.
- Team et al. [2025] Gemini Robotics Team, Krzysztof Choromanski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, Sean Kirmani, Isabel Leal, et al. Evaluating gemini robotics policies in a veo world simulator. CoRR, abs/2512.10675, 2025.
- Unterthiner et al. [2018] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. CoRR, abs/1812.01717, 2018.
- Wang et al. [2025] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
- Wang et al. [2024] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. In European Conference on Computer Vision (ECCV), 2024.
- Wang et al. [2026] Yixuan Wang, Rhythm Syed, Fangyu Wu, Mengchao Zhang, Aykut Onol, Jose Barreiros, Hooshang Nayyeri, Tony Dear, Huan Zhang, and Yunzhu Li. Interactive world simulator for robot policy training and evaluation. CoRR, abs/2603.08546, 2026.
- Wiedemer et al. [2025] Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are zero-shot learners and reasoners. CoRR, abs/2509.20328, 2025.
- Wissler [1905] Clark Wissler. The spearman correlation formula. Science, 22(558):309–311, 1905.
- Wu et al. [2023] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning (CoRL), 2023.
- Wu et al. [2025] Yilin Wu, Ran Tian, Gokul Swamy, and Andrea Bajcsy. From foresight to forethought: Vlm-in-the-loop policy steering via latent alignment. In Robotics: Science and Systems (RSS), 2025.
- Yin et al. [2026] Tenny Yin, Zhiting Mei, Zhonghe Zheng, Miyu Yamane, David Wang, Jade Sceats, Samuel M Bateman, Lihan Zha, Apurva Badithela, Ola Shorinwa, et al. Playworld: Learning robot world models from autonomous play. CoRR, abs/2603.09030, 2026.
- Zhang and Sennrich [2019] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Neural Information Processing Systems (NeurIPS), 2019.
- Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
- Zhou et al. [2025] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. In International Conference on Machine Learning (ICML), 2025.
Contents
Appendix A1 Robot Setup & Tasks
A1.1 Tasks Details
We collect real-world finetuning data from on five manipulation tasks, with 50 rollouts per task on our DROID setup as shown in Fig. 10. We select tasks for which the base policy achieves at least success, ensuring that the collected rollouts contain both successful and failed executions while remaining within the policy’s competence. The tasks are designed to cover a diverse set of manipulation regimes, including rigid-object pick-and-place, deformable-object manipulation, and dynamic manipulation as shown in Fig. 10.
Stack Bowls requires the robot to stack one bowl on top of another. Two bowls are randomly placed on the table, and the robot must place bowl on bowl , where and .
PnP Bag requires the robot to pick up a bag of chips and place it on a green plate. We use two types of chip bags and randomly sample one in each episode. The bag is deformable, making the grasp outcome and object motion difficult to predict.
PnP Marker requires the robot to pick up an Expo marker lying horizontally on the table and place it inside a container. The marker color is randomly selected from black and orange, and the target container is randomly selected from a paper cup and a blue mug. This task requires precise grasping and large end-effector reorientation to insert the marker vertically.
PnP Towel requires the robot to pick up a towel and place it into a basket. We use two towel variants, a folded thick red kitchen towel and a thin gray square towel, and two basket variants, orange and blue. The task is challenging because the towel is deformable, and its resulting shape depends strongly on the grasp location and, for the folded towel, the number of layers grasped.
Pour Beans requires the robot to pick up a cup containing coffee beans and pour them into a blue bowl. This task tests dynamic manipulation, as the granular motion of the beans is difficult to predict and successful execution requires accurate control of cup pose, pouring angle, and motion to avoid spilling outside the bowl.
A1.2 Action Space
The base policy on the DROID setup outputs joint-velocity commands for control. To match this action representation, we define the action space of our world model in joint space, avoiding potential compounding errors from converting actions into alternative representations such as Cartesian space. However, we find that directly conditioning the world model on joint velocities leads to lower generation quality. Therefore, following prior work [12], we use a lightweight action adapter to convert joint velocities into joint positions.
For policy evaluation, because the joint positions at the end of each trajectory are already available, we directly use joint positions as inputs to the world model. During test-time planning, however, predictions must be made from the joint-velocity actions proposed by the policy. We therefore use the trained action adapter to predict the corresponding joint positions, and condition the world model generation on these adapted actions. The following section describes the details of the action adapter.
A1.2.1 Action Adapter
Overview.
The action adapter is a lightweight feedforward module that bridges the world model’s action representation (joint velocity commands and binary gripper signals) and the robot’s observable state (absolute joint positions and gripper width). Given the robot’s current state and a chunk of actions produced by the world model, it predicts the resulting sequence of joint-position and gripper-position deltas, which are then integrated to obtain future absolute states.
Input representation.
The model receives two groups of inputs:
-
State token: the current 7-DOF joint position concatenated with the current scalar gripper position, forming a -dimensional vector.
-
Action tokens: a chunk of joint-velocity commands () concatenated with gripper actions (), flattened to dimensions.
Both groups are concatenated into a single input vector of dimension (for ).
Architecture.
The adapter is a three-layer MLP with hidden size 512 and SiLU activations:
The output is reshaped to and split into predicted joint deltas () and predicted gripper deltas ().
Normalization.
All continuous inputs and targets are min-max normalized to using per-dimension 1st/99th-percentile bounds computed from the training set, which is more robust to outliers than global min/max. Gripper action commands are binarized (, otherwise ) prior to input, reflecting their discrete open/close semantics.
Loss function.
The model is trained with a weighted MSE loss on the normalized delta targets:
The gripper is up-weighted because it has a much smaller dynamic range than the joint dimensions and would otherwise be under-penalized relative to its importance in grasp and place predictions.
Inference.
At test time the model predicts , denormalizes them, and integrates from the current state:
Training details.
The adapter is trained for 15 epochs using Adam (, batch size 128) on 50 hours of proprioceptive teleoperation data. Each training sample consists of a randomly drawn window of consecutive timesteps from an episode; the first timestep provides the current state and the remaining timesteps provide the action chunk and delta targets.
Appendix A2 Implementation Details
A2.1 Architecture Details
We use the VAE from Stable Diffusion 3 [9] to encode 19032 image frames from camera views into the latent space. Our efficient transformer architecture is a 32-layer transformer with 1536 hidden dimensions and 16 attention heads. Each layer comprises of a spatial layer that attends to all the patches in and a causal temporal layer to attend over patches from prior observations. The actions and proproceptive states are normalized using statistics obtained with the training dataset of DROID. We obtain the reward annotations from Robometer [25] and use the progress rewards to train the reward head and the critic. The reward and critic networks uses an AdaPool [5] layer to compress the tokens to a vector, and has MLP layers with this vector concatenated with the CLIP embedding [34] (provides a representation for the language instruction).
A2.2 Training Details
WEAVER Pretraining.WEAVER has 928M paramters in total and is trained for 1M gradient steps on 4H100 GPUs for 10 days. The pretraining is done on the DROID dataset [41]. We also maintain an exponential moving average (EMA) of model weights during training with . We use a learning rate warmup for the initial 10000 steps and keep a constant learning rate of post warmup. We provide the hyperparameters in Table 2.
WEAVER Finetuning. For complex tasks like pouring that is underrepresented in the pretraining dataset, the world model is inaccurate at predictions. To mitigate covariate shift and improve generations, a potential solution is to finetune the world model with task dataset. In this work, we finetune WEAVER on a small datasets of 250 trajectories (50 for each task) collected using the VLA. We finetune WEAVER for 16K gradient steps using a smaller learning rate of . Other hyperparameters are similar to pretraining (as described in Table 2) and the training takes 6 hours on 4H100 GPUs.
Finetuning. We follow the original setup of the droid finetuning configuration in openpi to finetune our policy. We use the open-sourced pre-trained droid checkpoint as the base policy. Normalization statistics are inherited from the original DROID checkpoint and held fixed throughout fine-tuning to preserve compatibility with the pretrained observation encoder. All runs with dataset size smaller than 5000 segments use a batch size of 32, a peak learning rate of with cosine decay and no warmup, and language instructions sourced from task annotations. For datasets of size trajectories, we fine-tune for 10,000 gradient steps with a cosine decay over 10,000 steps and warmup steps of 1000 and peak learning rate of 2.5e-5. For smaller datasets (1,000–2,000 trajectories), we reduce training to mitigate overfitting.
Reward Labeling.We label each trajectory with a per-frame progress reward using the Robometer evaluation server [25]. For each episode, we extract frames from the recorded video and downsampled to 1 fps using, selecting the right-camera view from the DROID setup. The sampled frames and the episode’s language instruction are sent in a single forward pass to the Robometer eval server, which returns a per-frame progress prediction representing the estimated fraction of task completion at frame , along with an optional per-frame success probability .
Because frames are subsampled at 1 fps, the resulting reward sequence is shorter than the original video. We realign rewards to the full video length by linear interpolation: letting denote the original frame count and the number of inferred frames, we place sampled values at positions and interpolate onto the integer grid . We choose reward_progress (the interpolated progress signal) as our final reward annotation because it is more aigned with actual task outcome. We substract the reward progress by -1 to make the reward fall in as labels for training.
| Name | Value |
|---|---|
| World Models | |
| Layers | 32 |
| Heads | 16 |
| Embedding dimension | 1536 |
| Head dimension | 96 |
| SPRINT probability | 0.5 |
| Reward Model and Critic | |
| MLP layers | 2 |
| Discount factor | .995 |
| Return lambda | .95 |
| Pretraining | |
| Batch size | 32 |
| Batch length | 8 |
| Memory frames (p) | 6 |
| Memory frame stride (m) | 5 |
| Optimizer | AdamW |
| Proprioceptive State loss scale () | 0.1 |
| LR | |
| Warmup steps | 10000 |
| EMA decay | .9999 |
| Training Steps | 1000000 |
| Finetuning | |
| Batch size | 32 |
| LR | |
| Warmup steps | 200 |
| EMA decay | .9999 |
| Training Steps | 16000 |
A2.3 Inference
Inference noise schedules. We evaluate several deterministic schedules to map discrete inference index to the noise level , where is the number of denoising steps. We describe the different noise schedules compared– linear, sigmoid, power and cosine:
where is the logistic sigmoid and controls the sharpness. For the sigmoid schedule, we normalize endpoints to be and . The linear schedule allocates steps uniformly, cosine and power allocate more budget near low-noise regions, and sigmoid concentrates updates around the middle of the trajectory.
Rectified-Flow. To further reduce inference time and NFE for downstream tasks like test-time planning, we used ReFlow [28] to post-train WEAVER-FT model, and call it WEAVER-ReFlow. The teacher and student model are initialised with a WEAVER-FT model where we freeze the teacher model. At each training iteration, we sample noise and predict future latents with the teacher model . This student model is updated with the predicted latent as the target using mean squared error loss given by: . The post-training with rectified flow is performed for 2K gradient steps on (6 hours) with a learning rate of .
Appendix A3 Additional World Model Evaluation Results
We provide additional results to evaluate WEAVER at coherent generations, impact of KV Cache on inference time, benefits of noise schedules, finetuning on task data, and post-training with rectified flow.
A3.1 World Model Evaluation
For a trajectory in validation dataset, we generated the rollout from the 20-th step, and compute the metrics using the generations for next 10s. We use the first 20 frames to initialize the memory and history for WEAVER and Ctrl-World. We report LPIPS [53], FID [20], and FVD [43] obtained using the ground truth videos. To obtain LPIPS, we utilize the functionality in torchmetrics222https://github.com/Lightning-AI/torchmetrics that uses the per-frame features obtained from vgg layers. To compute the FID, we use the implementation provided in pytorch-fid333https://github.com/mseitzer/pytorch-fid repository. Our results on FVD are computed using the Style-GAN-V444https://github.com/universome/stylegan-v repository. Here, we subsample multiple trajectories of 16 frames with a stride of 8 from each trajectory.
| Exterior | Wrist | |||||||
| Method | NFE | LPIPS | FID | FVD | LPIPS | FID | FVD | Time (s) |
| \rowcolorpastellavender DROID | ||||||||
| Ctrl-World | 8 | 0.169 | 31.63 | 116.14 | 0.407 | 52.40 | 347.69 | 8.14 |
| 16 | 0.165 | 26.09 | 78.73 | 0.392 | 33.83 | 195.37 | 14.65 | |
| 32 | 0.168 | 23.63 | 63.55 | 0.389 | 27.14 | 114.87 | 27.67 | |
| 50 | 0.168 | 22.44 | 55.05 | 0.388 | 25.32 | 91.77 | 42.33 | |
| WEAVER | 8 | \cellcolorlightblue0.117 | 10.59 | 28.97 | \cellcolorlightblue0.372 | 24.25 | 104.53 | \cellcolorlightblue2.53 |
| 16 | \cellcolorlightblue0.117 | 10.20 | 27.83 | \cellcolorlightblue0.371 | 21.50 | 90.72 | 4.78 | |
| 32 | \cellcolorlightblue0.120 | \cellcolorlightblue 9.67 | \cellcolorlightblue25.94 | \cellcolorlightblue0.378 | \cellcolorlightblue17.53 | \cellcolorlightblue63.36 | 9.22 | |
| 50 | 0.122 | \cellcolorlightblue9.51 | \cellcolorlightblue26.54 | \cellcolorlightblue0.378 | \cellcolorlightblue16.75 | 66.89 | 14.25 | |
| \rowcolorpastelmint New dataset | ||||||||
| Ctrl-World | 8 | 0.193 | 48.90 | 226.29 | 0.374 | 51.26 | 434.84 | 8.14 |
| 16 | 0.182 | 36.16 | 139.54 | 0.366 | 38.76 | 277.13 | 14.65 | |
| 32 | 0.183 | 32.18 | 105.38 | 0.365 | 33.73 | 173.15 | 27.67 | |
| 50 | 0.184 | 31.44 | 91.48 | 0.367 | 33.47 | 145.86 | 42.33 | |
| WEAVER | 8 | \cellcolorlightblue0.154 | \cellcolorlightblue23.89 | 89.55 | \cellcolorlightblue0.364 | 31.70 | 193.55 | \cellcolorlightblue2.53 |
| 16 | \cellcolorlightblue 0.155 | \cellcolorlightblue23.95 | 88.27 | \cellcolorlightblue0.364 | 30.77 | 184.62 | 4.78 | |
| 32 | \cellcolorlightblue0.157 | \cellcolorlightblue23.45 | 92.36 | \cellcolorlightblue0.365 | 28.24 | \cellcolorlightblue148.85 | 9.22 | |
| 50 | \cellcolorlightblue0.159 | \cellcolorlightblue23.48 | \cellcolorlightblue87.03 | \cellcolorlightblue0.371 | \cellcolorlightblue27.37 | \cellcolorlightblue145.04 | 14.25 | |
A3.2 Quantitative Results
Table A3.1 reports the comparison of WEAVER and Ctrl-World on DROID and OOD datasets across multiple metrics. We observe that performance of Ctrl-World deteriorates with lower NFE whereas WEAVER shows slight drop in performance with decrease in NFE. Moreover, with similar NFE values, our method is faster at generating rollouts than Ctrl-World. In Fig. 11, we present the comparison of FID and inference time and observe that WEAVER with lowe NFE of 8 outperforms Ctrl-World with large NFE of 50. We also include qualitative results of different NFEs with different world models in Fig. 14 and Fig. 15.
| / NFE | 8 | 16 | 32 | 50 |
|---|---|---|---|---|
| Without KV cache | 3.30 | 6.54 | 12.98 | 20.21 |
| With KV cache | 2.53 | 4.78 | 9.22 | 14.25 |
A3.3 Reducing inference time with KV Cache
During the iterative denoising process, the latents of memory and history frames are passed with a constant noise level . Since it does not change during this process, we compute the cache of key-value vectors of memory and history latents at the first denoising step. In Table 4, we report that KV Cache can reduce inference time by up to 30%.
| Exterior | Wrist | ||||||
| Method | Schedule | LPIPS | FID | FVD | LPIPS | FID | FVD |
| \rowcolorpastellavender DROID | |||||||
| Ctrl-World | linear | 0.165 | 26.09 | 78.73 | 0.392 | 33.83 | 195.37 |
| WEAVER | linear | \cellcolorlightblue0.117 | 11.32 | \cellcolorlightblue26.38 | \cellcolorlightblue0.375 | 24.43 | 98.82 |
| sigmoid | \cellcolorlightblue0.117 | \cellcolorlightblue10.88 | 29.76 | \cellcolorlightblue0.375 | 22.69 | 104.89 | |
| power | \cellcolorlightblue0.117 | \cellcolorlightblue10.57 | 27.93 | \cellcolorlightblue0.369 | \cellcolorlightblue21.17 | \cellcolorlightblue91.51 | |
| cosine | \cellcolorlightblue0.117 | \cellcolorlightblue10.20 | 27.83 | \cellcolorlightblue0.371 | \cellcolorlightblue21.50 | \cellcolorlightblue90.72 | |
| \rowcolorpastelmint New dataset | |||||||
| Ctrl-World | linear | 0.182 | 36.16 | 139.54 | \cellcolorlightblue0.366 | 38.76 | 277.13 |
| WEAVER | linear | \cellcolorlightblue0.157 | 25.37 | 96.16 | \cellcolorlightblue0.367 | 33.24 | 217.65 |
| sigmoid | \cellcolorlightblue0.156 | 24.83 | 93.15 | \cellcolorlightblue0.367 | 32.48 | 216.89 | |
| power | \cellcolorlightblue0.155 | \cellcolorlightblue23.82 | \cellcolorlightblue84.91 | \cellcolorlightblue0.363 | \cellcolorlightblue31.60 | \cellcolorlightblue185.30 | |
| cosine | \cellcolorlightblue0.155 | \cellcolorlightblue23.95 | 88.27 | \cellcolorlightblue0.364 | \cellcolorlightblue30.77 | \cellcolorlightblue184.62 | |
A3.4 Noise schedules during inference
Table A3.3 compares different noise schedules where we observe that both power and cosine noise schedules perform better than sigmoid and linear noise schedules. Since the world model needs to generate with higher fidelity, the noise schedules that allocate more bandwidth at low noise regions aids in generating fine-grained details.
| Exterior | Wrist | ||||||
| Method | NFE | LPIPS | FID | FVD | LPIPS | FID | FVD |
| \rowcolorpastelmint Task Data (OOD) | |||||||
| Ctrl-World | 50 | 0.184 | 31.44 | 91.48 | 0.367 | 33.47 | 145.86 |
| Ctrl-World-FT | 16 | 0.140 | 29.89 | 81.16 | 0.295 | 34.77 | 283.00 |
| 50 | 0.142 | 25.96 | 58.27 | 0.292 | 25.80 | 134.69 | |
| WEAVER | 50 | 0.159 | 23.48 | 87.03 | 0.371 | 27.37 | 145.04 |
| WEAVER-FT | 4 | 0.116 | 16.62 | 50.68 | 0.304 | 33.32 | 219.15 |
| 16 | 0.118 | 14.24 | 40.50 | 0.303 | 23.39 | 146.35 | |
| 50 | 0.121 | 13.69 | 40.09 | 0.308 | 18.73 | 99.81 | |
| WEAVER-REFLOW | 4 | 0.123 | 14.95 | 44.30 | 0.312 | 23.56 | 138.67 |
A3.5 Finetuning
In Table A3.4, we observe that finetuning significantly improves performance and with NFE=16 it performs better than WEAVER with NFE=50. To provide a fair comparison with Ctrl-World, we finetune the baseline for 20K gradient steps on and see that the finetuned Ctrl-World (called Ctrl-World-FT) performs better than the pretrained model. However, WEAVER-FT ourperforms Ctrl-World-FT across metrics and the performance is still larger with low NFE=16. This further demonstrates that finetuning does not help in reducing inference time for Ctrl-World. We also provide qualitative results of the rollouts generated from Ctrl-World, WEAVER, WEAVER-FT in Fig. 16, Fig. 17 and Fig. 18.
A3.6 Posttraining with Rectified Flow
In table A3.4, we present the results of WEAVER-ReFlow with small inference budget and observe that it reduces the performance gap with WEAVER-FT evaluated with a large NFE=16. This makes it suitable for test-time steering as observed in Section 3.2.
| Component | Notation | Batch Size | Horizon | Runtime (s) |
| Policy sampling | – | |||
| Dynamics model | 4 | 9 | ||
| (WEAVER) | 4 | 12 | ||
| 4 | 15 | |||
| 1 | 15 | |||
| Dynamics Model | 4 | 15 | ||
| (Ctrl-World) | 1 | 15 | ||
| Reward inference | 4 | – | ||
| Critic inference | 4 | – |
Appendix A4 Additional Downstream Application Results
A4.1 Policy Evaluation Results
We provide the full policy evaluation rollouts in Fig. 12. Both Ctrl-World and WEAVER struggle to accurately predict policy performance, especially on challenging tasks involving dynamic manipulation, such as pouring beans, and deformable object manipulation, such as bag and towel manipulation. For the PnP Bag task, grasping the bag is particularly challenging because the world model must accurately infer the gripper depth across two camera views while also modeling the contact dynamics between the gripper and the deformable object. These challenges become more pronounced as the prediction horizon increases. In contrast, WEAVER-FT substantially improves evaluation accuracy through finetuning as shown in Fig. 19 and Fig. 20. Future work could further improve long-horizon prediction by designing better memory and history representations, enabling the model to better reason about occlusions and deformable object dynamics.
In addition to Pearson correlation and MMRV, we also report RMSE and Spearman rank correlation [48]. Across these metrics, we observe a consistent trend: WEAVER-FT achieves the strongest correlation and lowest prediction error. In addition, WEAVER outperforms Ctrl-World in zero-shot policy evaluation on out-of-distribution task dataset. The full quantitative results are shown in Table 8.
| Method | RMSE | Spearman | Pearson | MMRV |
|---|---|---|---|---|
| CtrlWorld | 0.410 | 0.523 | 0.552 | 0.215 |
| Ours | 0.359 | 0.594 | 0.563 | 0.155 |
| Ours-FT | 0.188 | 0.870 | 0.863 | 0.035 |
A4.2 Policy Improvement Results
We provide additional qualitative results of policy improvement in Fig. 13. These examples show that the base policy often suffers from imprecise grasping and placement, as well as insufficient adjustment during dynamic manipulation. We also observe that the base policy tends to produce larger per-step motions, resulting in unstable robot control. In contrast, the finetuned policy substantially reduces these large movements and sharpens the action distribution, leading to smoother and more stable execution.
We also note that the RoboMeter reward labels are not perfect. For the PnP Marker task, we observe cases where the reward model fails to distinguish fine-grained placement accuracy, which can introduce noise into the predicted rewards. Future work could improve reward supervision by collecting more diverse failure data to train a more general and precise reward model. To mitigate the effect of noisy reward labels, we set the advantage threshold to , which helps prevent low-quality segments from being selected for finetuning and potentially degrading policy performance. As shown in Fig. 4, our filtering procedure is able to select the best action samples among the candidates.
A4.3 Test-Time Planning Results
Inference-Time Latency. Table 7 reports the inference-time breakdown of test-time planning on an A6000 Ada GPU. Overall, the runtime is dominated by the dynamics model imagination, while reward and critic inference are negligible, taking less than s each. For WEAVER, the dynamics runtime increases moderately with the imagination horizon: from s at horizon , to s at horizon , and s at horizon with batch size . Including policy sampling, reward inference, and critic inference, the full planning latency is approximately s, s, and s for horizons , , and , respectively.
Compared to Ctrl-World, WEAVER substantially reduces latency during imagination. At horizon and batch size , WEAVER takes s for dynamics prediction, while Ctrl-World requires s, corresponding to a speedup. The same trend holds at batch size , where WEAVER takes s compared to s for Ctrl-World, yielding a speedup. These results show that WEAVER enables substantially lower-latency test-time planning, making repeated world-model imagination practical during policy execution.
Appendix A5 Limitations
While WEAVER demonstrates the promise of large-scale world models for policy evaluation, policy improvement, and test-time planning, several limitations remain.
A5.1 Partial Observability
Our world model relies primarily on visual observations, which provide only partial access to the underlying physical state. During manipulation, task-relevant information such as object contacts, grasp stability, applied forces, or occluded object geometry may be hidden from all available camera views. This limitation is especially pronounced for wrist-camera observations, where the viewpoint changes continuously, and for cluttered scenes where objects may leave the field of view or become occluded by the gripper. Although memory and multi-view conditioning mitigate this issue, purely visual prediction may still fail when the missing state cannot be inferred from image history alone. Incorporating additional sensing modalities, such as tactile feedback, force-torque sensing, or depth, may improve state estimation and long-horizon prediction under occlusion.
A5.2 Complex Deformable and Dynamic Interactions
Deformable-object manipulation and dynamic manipulation remain challenging for learned world models. Objects such as towels, bags, and granular materials exhibit high-dimensional, history-dependent dynamics that are difficult to capture from limited robot data. Small errors in predicted contact, grasp location, or object configuration can compound over time and lead to qualitatively incorrect rollouts. This is particularly evident in tasks such as pouring, where the motion of granular material depends sensitively on cup pose, velocity, and contact with the container. Future work may improve prediction fidelity by incorporating physics priors, hybrid neural-physics models, or neural simulators specialized for deformable and granular dynamics.
A5.3 Limited Planning Horizon at Test Time
Although our inference acceleration strategies make test-time planning feasible with a large generative world model, latency still limits online planning to a single action chunk. As a result, the planner can improve near-term action selection but cannot yet perform long-horizon lookahead. This restricts its ability to reason about delayed consequences or multi-stage recovery behaviors. Further improvements in sampling efficiency, model distillation, value estimation, or hierarchical planning could enable longer-horizon online reasoning while maintaining real-time control.
A5.4 Data Coverage and Embodiment Diversity
Our world model is pretrained primarily on DROID, which provides large-scale robot interaction data but is still tied to a specific robot embodiment and data collection setup. This may limit generalization to substantially different robots, camera configurations, end-effectors. In addition, some task dynamics in our evaluation, such as granular pouring, are underrepresented in the pretraining data. Scaling world-model training to more diverse sources, including cross-embodiment robot datasets, simulation data, and human videos, may improve robustness and broaden the range of behaviors that can be accurately imagined.
A5.5 Noisy Reward Supervision
Our latent reward and critic heads are trained using labels from an off-the-shelf reward model. While this enables efficient latent-space evaluation, the resulting supervision can be noisy or incomplete, especially for subtle failure modes. For example, a reward model may fail to distinguish between visually similar but semantically different outcomes, or may be insensitive to small errors in contact, placement, or task completion. Such noise can affect both policy evaluation and downstream policy improvement. A more reliable reward model trained on large-scale robot success and failure data, potentially with calibrated uncertainty, would likely improve the reliability of imagined rollout evaluation.
Overall, these limitations suggest that future progress will require not only larger and faster world models, but also richer sensing, broader data coverage, stronger physical inductive biases, and more accurate reward supervision.