世界模型自蒸馏:训练世界模型解决通用任务
阅读原文· arxiv.org提出结合自蒸馏与强化学习的可扩展框架,激发预训练视频扩散模型(Demonstrator)的任务解决能力。给定未标注场景图像,视觉语言模型(VLM)生成候选任务及详细步骤,条件化Demonstrator生成视频;通过蒸馏将执行知识迁移至仅以图像和简短任务提示为条件的Executor,无需配对任务-视频数据。进一步利用VLM反馈的强化学习优化Executor。在WorldTasks-Benchmark和DreamGen机器人基准上,Executor在VLM评估协议下超越Demonstrator,并有效迁移至机器人任务。
Pretrained video generators are promising visual world models that exhibit emergent task-solving abilities; however, their reliance on detailed textual descriptions limits their direct use for planning and decision-making. Existing approaches either outsource this reasoning to language or vision-language models, or rely on supervised fine-tuning with paired task-execution videos, which are costly to collect and difficult to scale. We propose a scalable framework that elicits task-solving ability in such models by combining self-distillation with reinforcement learning. Given an unlabeled scene image, a vision-language model generates a candidate task and a detailed step-by-step solution. The solution conditions a pretrained video diffusion model, the Demonstrator; we distill its behavior into an Executor conditioned only on the image and a short task prompt. This transfers execution knowledge from caption-guided generation to instruction-conditioned task solving without curated task-video supervision. We further improve the Executor with reinforcement learning from VLM feedback, exploiting the asymmetry between judging whether a sampled video satisfies a task and generating the solution. Experiments on our proposed WorldTasks-Benchmark and the DreamGen robotics benchmark show that the Executor surpasses the Demonstrator under our VLM-based evaluation protocol and transfers competitively to robotic tasks.