郑凯文
何冠德
赵敏
张金涛
陈华宇
陈剑飞
林辰轩
刘明宇
朱军
马千里
德克萨斯大学奥斯汀分校
zkwthu@gmail.com;
dcszj@tsinghua.edu.cn;
mingyul@nvidia.com
\abscontent
基于因果扩散Transformer的自回归视频扩散，已成为实时流式视频生成和动作条件交互式世界模型的主要范式。在本工作中，我们将先进的扩散蒸馏框架rCM扩展到自回归视频扩散领域。rCM的核心思想在于扩散蒸馏中前向散度和反向散度之间的互补性，前者由一致性模型（CM）代表，后者由分布匹配蒸馏（DMD）代表。这一思想自然地延续到自回归设定中，其中教师强制（TF）提供了一种离线、前向散度的因果训练范式，而自强制（SF）则对应一种在策略、反向散度的精炼过程。
我们的贡献在于：（1）通过大量实验，我们证明教师强制CM目前是自强制DMD作为初始化策略的最佳补充；（2）我们首次实现了基于教师强制的连续时间CM（例如sCM/MeanFlow）用于自回归视频扩散，这得益于我们定制的掩码FlashAttention-2 JVP内核，与离散时间CM（dCM）相比实现了10倍的收敛速度提升；（3）我们提出了Causal-rCM，这是一个领先、统一且可扩展的算法-基础设施开源方案，用于扩散蒸馏和因果训练；（4）我们在逐帧和逐块两种设定下均实现了最先进的流式视频生成性能，且仅使用合成数据进行训练。
值得注意的是，我们蒸馏后的2步因果Wan2.1-1.3B模型仅需1或2个采样步骤，就达到了84.63的VBench-T2V分数。我们进一步将Causal-rCM应用于Cosmos 3，这是一个面向物理AI的先进全模态世界基础模型，具备动作条件生成能力，从而实现了交互式世界模型。
1 引言
视频扩散模型被广泛视为一种世界模拟器（Brooks等人，2024；Bao等人，2024；Kong等人，2024；Wan等人，2025；Ali等人，2025；Gao等人，2025；Seedance等人，2026；NVIDIA，2026）。与使用双向注意力扩散Transformer对所有帧联合去噪不同，自回归（AR）视频扩散（Jin等人，2025；Teng等人，2025；Chen等人，2025a）采用因果注意力扩散Transformer进行下一帧或下一块的预测。这反映了离散扩散领域中从掩码扩散（Sahoo等人，2024；Shi等人，2024；Zheng等人，2025c）到块扩散（Arriola等人，2025）的转变。在该范式中，模型在帧或块之间是自回归的，而扩散去噪则在每个帧或块内部进行。这使得流式长视频生成（Huang等人，2025b；Yang等人，2026；Chen等人，2026）、交互式世界模型（Hong等人，2025；HunyuanWorld，2025；He等人，2025；Robbyant Team等人，2026）以及用于闭环机器人控制的具身AR视频扩散（Feng等人，2025；Li等人，2026c；Ye等人，2026）成为可能。
常见的因果训练范式，如教师强制（TF）和扩散强制（DF）（Chen等人，2024），在AR扩散推理过程中会随时间出现误差累积和质量下降，这通常被称为曝光偏差（Schmidt，2019；Ning等人，2024）。最近的自强制范式（Huang等人，2025a；Lin等人，2025b）通过使用在线策略训练来解决训练-推理差距，并结合分布匹配蒸馏（DMD）（Yin等人，2024b, a）或对抗性GAN损失（Lin等人，2025a）进行扩散步蒸馏，从而解决了这一问题。自强制方法已推动AR视频扩散在流式和交互式场景中向实用的低延迟、实时和长时程生成方向发展。
然而，使用 DMD 或 GAN 目标函数的自强制方法对初始化敏感，且容易出现模式坍塌，因为 DMD 风格的目标函数基于反向 KL 散度，并优化学生模型生成的轨迹。因此，现有的自回归扩散系统在自强制之前引入了不同的初始化策略，例如 ODE 对回归（Yin 等人，2025；Huang 等人，2025a；He 等人，2025；Zhu 等人，2026）、扩散强制风格的因果适应（Huang 等人，2025b；Robbyant Team 等人，2026），或混合 TF/DF 初始化（Hong 等人，2025）。这些设计表明，在策略内分布匹配之前，一个稳定的离线因果目标至关重要，但初始化、因果训练范式与蒸馏损失之间的联系仍有待探索。
在本工作中，我们引入了 Causal-rCM，将 rCM（分数正则化一致性模型）（Zheng 等人，2025d）扩展到自回归视频扩散领域。在 rCM 中，关键洞察在于蒸馏目标层面上的前向-反向互补性：一致性模型作为前向散度、轨迹保持的目标，而 DMD 作为反向散度、分布匹配的目标。在自回归扩散中，因果训练范式层面出现了类似的互补性：教师强制提供了一种离线的、覆盖模式的训练信号，而自强制则在自回归轨迹生成下提供了一种策略内的精炼信号。基于这种对应关系，Causal-rCM 使用教师强制的一致性模型，在离线因果上下文和教师轨迹上进行少步因果蒸馏，并使用自强制 DMD 直接优化推理时的少步分布。
与先前工作的关系
基于一致性模型的方法被广泛用作基于 DMD 和 GAN 的扩散蒸馏的初始化或正则化手段（Lin 等人，2025a；Zheng 等人，2025d）。值得注意的是，对于自回归扩散，APT2（Lin 等人，2025b）已采用基于教师强制的一致性模型作为自强制阶段的初始化，后续通过帧级单射性视角获得了理论支持（Zhu 等人，2026）。Causal-rCM 与先前工作的不同之处在于：(1) 提供了关于不同因果训练范式、蒸馏损失及其协同效应的统一散度视角，呼应了 rCM 的高级原则；(2) 对自强制 DMD 的不同初始化策略进行了全面系统的研究，揭示了各自的优缺点；(3) 首次实现了基于教师强制的连续时间一致性模型（sCM（Lu 和 Song，2024），MeanFlow（Geng 等人，2025）），并采用我们的自定义掩码 FlashAttention-2 JVP 内核，与离散时间一致性模型（dCM）相比实现了 10 倍的收敛速度提升；(4) 引入了一套领先、统一且可扩展的算法-基础设施开源方案，用于扩散蒸馏和因果训练，在自回归扩散蒸馏中实现了最先进的性能。
前向-反向目标互补性
| 方法 | 领域 | 前向组件（预训练 / 离线） | 反向组件（后训练 / 在线策略） | 效果 / 要点 |
| DDO（Zheng 等人，2025b） | 扩散 / 自回归中期训练 | 对真实数据的扩散损失 | 对自生成负样本的反似然扩散损失 | 在无需辅助数据/模型的情况下，在 ImageNet 上取得了新的 FID 记录 |
| DiffusionNFT（Zheng 等人，2025a） | 扩散强化学习 | 前向过程扩散目标 | 基于奖励排序的正/负生成样本 | 25 倍效率提升 |
| DDRL（Ye 等人，2025） | 扩散强化学习 | 对离线数据的前向 KL 散度 / 扩散损失正则化 | 对生成的轨迹进行 GRPO 风格的奖励优化 | 缓解奖励破解和多样性崩溃 |
| rCM（Zheng 等人，2025d） | 扩散蒸馏 | 对数据/教师轨迹的（s）CM 损失 | 对学生生成样本的 DMD 损失 | 缓解模式崩溃 |
| Causal-rCM | 自回归扩散蒸馏 | 在离线因果上下文上的教师强制一致性模型 | 在自回归学生模型 rollout 上的自强制 DMD | TF-CM 通过因果结构和模式覆盖来初始化 SF |
注。这种互补性既可以在单个联合阶段中实现，也可以按正向到反向的顺序在多个独立阶段中实现。我们使用“在策略”来强调自生成的样本或 rollout；在扩散强化学习中，此类数据可以是在线的，但在严格的强化学习意义上属于离策略。
联合利用正向和反向目标的更广泛理念已出现在扩散中期训练、扩散强化学习和扩散知识蒸馏中。正向或离线目标，例如扩散损失、教师强制损失以及基于真实数据或教师轨迹的一致性模型损失，可提供稳定的训练信号并保持模式覆盖。反向或在策略目标，例如 DMD、对抗性损失以及基于生成样本的奖励驱动优化，可直接改善生成分布，但对初始化和覆盖更为敏感。如表 1 所示，近期方法包括 DDO (Zheng et al., 2025b)、DiffusionNFT (Zheng et al., 2025a)、DDRL (Ye et al., 2025) 和 rCM (Zheng et al., 2025d) 均受益于这种互补性。Causal-rCM 在自回归扩散蒸馏中实例化了相同的原理：教师强制一致性模型作为正向/离线组件，而自强制 DMD 则作为反向/在策略组件。
2 背景
2.1 扩散模型
扩散模型（DM）（Ho 等人，2020；Song 等人，2020）通过用高斯噪声逐步扰动干净数据来学习连续数据分布，这一过程会生成一条轨迹及其对应的边缘分布，然后学习逆转这一过程。前向过程遵循一个具有预定义噪声调度表的闭式转移核，从而能够通过重参数化实现。扩散模型的采样过程可以遵循概率流常微分方程（PF-ODE），其中，且为得分函数（Song 等人，2020）。扩散模型的一个关键特性是不同参数化方式在理论上的等价性：网络可以预测得分（）、噪声（）、干净数据（）或速度（），且最优预测器之间可以通过解析方式相互转换（Zheng 等人，2023b）。采用速度参数化（Zheng 等人，2023b）时，扩散模型通过最小化均方误差（MSE）进行训练，其中回归目标为（记为），而 PF-ODE 简化为，这通常被称为流匹配（Lipman 等人，2022）。一个值得注意的特例是整流流（RF）（Liu 等人，2022），它采用调度表，从而将速度目标简化为。
2.2 扩散蒸馏
一致性蒸馏
一致性模型（CM）（Song 等人，2023）旨在学习一个一致性函数，该函数将教师 PF-ODE 轨迹上任意时刻的点映射到初始点。给定一个自由形式的学生网络，一致性函数通常参数化为，其中且（例如，在 RF 调度表下）。这种参数化自然满足边界条件。这里，是扩散模型中数据预测器（去噪器）的直接对应物，而则对应于速度预测器。
一致性模型的目标是在相邻时间步以及沿教师轨迹上强制学生输出保持一致。离散时间一致性模型（dCM）通过以下目标函数进行最小化，其中：
| （1） |
其中是正权重函数，是距离度量，是停止梯度版本的，而通过使用数值求解器从到求解教师 PF-ODE 得到。
连续时间一致性模型（sCM）（Lu and Song, 2024）对 dCM 取极限，以获得更精确的目标函数。当……时，瞬时一致性模型损失变为……，其中……是……沿教师 ODE 轨迹……在……处的切线。该切线可通过前向模式自动微分（即雅可比向量积，JVP）高效计算：……。sCM 进一步应用 MSE 重构和切线归一化，将损失简化为……
| (2) |
MeanFlow（Geng et al., 2025）可视为在整流流（RF）调度下将 sCM 与一致性轨迹模型（CTMs）（Kim et al., 2023）相结合。CTMs 通过增加另一个时间条件并定义一致性轨迹函数……来扩展一致性模型，该函数将点……映射到教师 ODE 轨迹上噪声更小的点……。从……到……的无穷小跳跃（即……）简化为扩散去噪器，并作为应用扩散损失的锚点。该锚点增强了训练稳定性，保留了多步采样能力，并支持从头训练少步模型。因此，CTMs 可视为扩散模型与一致性模型之间的插值。在连续时间情况下，CTMs 可通过与 sCM 类似的目标函数进行优化：
| (3) | ||||
在整流流（RF）调度下，我们有……，且……。由于……是速度预测器……，若将……视为真实速度……，则
| (4) |
这便还原了 MeanFlow 的目标函数。或者，我们可以设置……并使用相同的公式进行蒸馏，而非从头训练。
分布匹配蒸馏
分布匹配蒸馏（DMD）（Yin et al., 2024b, a）是一种简单有效的分数蒸馏方法（Wang et al., 2023; Zhou et al., 2024）。给定一个少步学生生成器……，其先验分布为……，DMD 旨在通过最小化学生分布与教师分布在其扩散边缘分布上的反向 KL 散度，来使学生分布……与教师分布……相匹配：
| (5) |
该目标函数的梯度可表示为学生分布与教师分布之间的分数差：
| (6) |
教师分数由预训练的扩散模型提供，而学生分数对于几步生成器而言是难以处理的。因此，DMD 在学生生成的样本上训练一个辅助的假分数网络，该网络作为学生分数的代理。在去噪器参数化中，分数差可以写成（吸收进一个与时间相关的标量后）假去噪器与教师去噪器之间的差值。借助 DMD 中的自适应归一化技巧，学生模型可以通过以下停止梯度均方误差目标进行更新：
| (7) |
DMD 在学生阶段和批评者阶段之间交替进行，形成一种类似于生成对抗网络的对抗训练动态。
2.3 自回归视频扩散
自回归视频扩散沿时间维度分解视频生成过程。给定一个被划分为帧或块（chunks）的视频隐序列，自回归模型在依赖先前块的条件下生成每个块：在每个时间块内部，模型仍然执行扩散去噪，例如在射频调度下，以速度为目标。与对所有帧进行联合去噪并使用全时间注意力的双向视频扩散不同，自回归视频扩散使用因果注意力，使得每个帧或块只关注过去的上下文。这使得模型能够像大语言模型一样使用 KV 缓存，并使其天然适用于流式生成和交互式生成。
图 3 展示了三种因果训练范式：教师强制、扩散强制和自强制。
在教师强制中，模型在关注干净的、真实的历史数据的同时预测当前带噪块，即。教师强制稳定且可通过特定的注意力掩码实现并行化，但它会造成训练-推理差距：在推理过程中，模型必须依赖自身生成的历史数据而非真实上下文作为条件。
DF 为不同的帧或块分配独立的噪声级别，并在块级因果注意力掩码下训练模型，即 。这使得模型能够接触带噪声的历史信息，从而提升鲁棒性。然而，训练与推理之间的差距依然存在：用合成噪声扰动真实视频，并不能匹配推理时模型自生成展开过程中累积的误差与伪影。
SF 在训练过程中直接模拟自回归推理。学生模型利用 KV 缓存依次展开块， ，损失函数作用于自生成的视频上。因此，SF 在模型自身的推理时上下文分布下进行训练，直接解决了 TF 和 DF 中因训练-推理差距导致的暴露偏差。SF 必须与反向类型的在策略目标函数（如 DMD 或 GAN 损失）结合使用。
3 Causal-rCM：一种领先、统一且可扩展的算法-基础设施开源方案，用于扩散蒸馏与因果训练
3.1 算法
为了将 rCM 扩展到自回归扩散，我们将其两个蒸馏目标（CM、DMD）分别与两种因果训练范式——教师强制（TF）和自强制（SF）——配对。这保留了 rCM 在自回归设定中的前向-反向对应关系：TF-CM 提供了一种离线、前向类型的一致性目标，而 SF-DMD 则提供了一种在策略、反向类型的分布匹配目标。
TF-CM 需要一个自回归扩散教师模型，该模型在基于 TF 的知识蒸馏过程中，需在与学生模型相同的干净上下文设置下进行评估。这种因果教师模型可以从零开始训练，也可以从预训练的双向扩散模型通过 TF 或 DF 适配而来。使用 TF 可以说更为合理，因为它能让教师模型接触到干净的过去帧，从而匹配基于 TF 的知识蒸馏所使用的上下文分布。CM 组件既可以实例化为简单的 dCM，也可以实例化为更先进的连续时间变体，例如 sCM 和 MeanFlow。对于 SF-DMD，我们遵循先前的工作（Huang 等人，2025a；Lin 等人，2025b），使用双向教师模型和双向伪分数网络，在自生成轨迹上提供真实分数和伪分数估计（图 3(c)），并应用 DMD 损失（公式 7）。
与以联合训练方式结合 CM 和 DMD 的 rCM 不同，Causal-rCM 按顺序依次应用 TF-CM 和 SF-DMD。整个流程包含三个阶段：(1) TF 将双向扩散模型转换为自回归扩散模型，该模型既作为后续 TF-CM 阶段的因果教师模型，也作为学生模型的初始化；(2) TF-CM 将因果教师模型蒸馏为少步因果学生模型，该学生模型作为后续 SF-DMD 阶段的学生模型初始化；(3) SF-DMD 精炼阶段进一步在学生模型自身的自回归轨迹上对其进行优化，从而缩小训练与推理之间的差距并减少曝光偏差。如图 4 总结所示，Causal-rCM 提供了一种简单而强大的方案，避免了繁琐的 ODE 对知识蒸馏（KD）（Luhman 和 Luhman，2021）和 GAN 风格的后训练，同时引入了一种新颖的 TF-sCM 实现，并达到了最先进的性能。
3.1.1 教师强制、教师强制 dCM 与自强制 DMD
基于 TF 训练的核心操作是将标准的单状态前向传播替换为一种打包的因果前向传播，该传播作用于拼接后的干净上下文和带噪目标。具体来说，对于一个速度预测器，我们不再评估 ，而是评估
| (8) |
在 TF 注意力掩码中，干净部分在时间步提供真实因果上下文，损失仅应用于噪声部分。该掩码确保每个噪声块仅关注其允许的干净历史及其自身的噪声 token，与图 3 中的 TF 模式相匹配。这种 TF 掩码注意力可以通过自定义掩码注意力算子（如 FlexAttention（Dong 等人，2024）或 MagiAttention（Zewei 和 Yunpeng，2025））来实现。另一种替代方案是两遍实现：首先在块因果注意力掩码下缓存干净 token，然后执行第二遍前向传播，其中噪声 token 关注缓存的干净上下文。然而，这种设计需要将干净 token 的 KV 缓存保留在计算图中，使其与激活检查点的兼容性较差，且内存占用更高。
在 RF 调度下的扩散回归目标中，普通的 TF 目标为
| (9) |
这给出了一个全步因果扩散模型。对于 TF-dCM，干净上下文保持固定，而噪声部分沿因果教师 PF-ODE 轨迹移动。设 为在相同 TF 掩码下，从 到 求解因果教师 ODE 得到的结果。学生模型最小化
| (10) | ||||
SF-DMD 在 TF-CM 之后应用。学生模型首先执行带 KV 缓存的时间自回归展开。在第 个块中，模型基于先前生成块的缓存状态生成当前干净块：
| (11) |
生成 后，通过一次缓存更新前向传播将其再次输入因果 Transformer，将其干净 token 的键/值状态追加到缓存中：
| (12) |
在每个块内， 通过从纯噪声 进行少步自展开去噪来实现：
| (13) |
在每次训练迭代中，模拟步数从 中随机采样。每个转换可以实例化为 CM 风格的反向去噪后接前向加噪，例如在 RF 调度下：
| (14) |
最终输出进入 DMD 损失函数。遵循标准做法（Yin 等人，2024a；Huang 等人，2025a），我们应用梯度截断使 SF-DMD 在内存使用上更高效。中间去噪步骤和先前块的 KV 缓存被分离（以 表示）。只有每个块的最终去噪步骤保持可微（以 表示），DMD 损失通过该步骤进行反向传播。
3.1.2 基于 JVP 的因果蒸馏与教师强制 sCM/MeanFlow
TF-sCM 使用与 TF 和 TF-dCM 相同的打包因果前向传播，但将有限步一致性目标替换为连续时间切线目标。干净上下文保持固定，而带噪 token 沿因果教师 ODE 移动。在 RF 调度下，定义带噪分支上的因果教师速度为
| (15) |
带噪分支上的 RF 一致性映射为
| (16) |
其沿因果教师轨迹的连续时间切线为
| (17) | ||||
此处，JVP 通过与原始预测相同的 TF 掩码打包前向传播计算。干净上下文的切线为零，只有带噪分支遵循教师速度。
那么 TF-sCM 的目标函数为
| (18) |
| (19) |
一个微妙但重要的设计选择是使用 sCM 的 RF 原生形式，而不是像 rCM（Zheng 等人，2025d）那样将 RF 速度模型包装进 TrigFlow 并应用 TrigFlow-sCM 目标函数。尽管不同的扩散噪声调度（如 TrigFlow 和 RF）在理论上可以通过时间相关缩放进行转换（Zheng 等人，2023b），但它们通常会导致 sCM 产生不同的归一化 MSE 目标函数（附录 A）。在双向设置中，rCM 发现 TrigFlow 包装有利于稳定性。然而，在我们的因果 TF 设置中，TrigFlow 包装的 TF-sCM 导致生成质量下降，而 RF 原生的 TF-sCM 则产生更平滑的输出。
3.1.3 扩展到带噪上下文和自定义步长调度
带噪上下文和自定义步长调度（Liu 等人，2026）是 AR 视频扩散蒸馏中最简单且最有效的两种推理加速技术。TF 和 SF 都可以自然地融入它们，如图 5 所示。
噪声上下文
与 LLM 不同，自回归视频扩散模型必须维护一个对去噪时间步敏感的 KV 缓存：标准的干净上下文自回归推理需要在每个分块的去噪步骤之后额外进行一次干净上下文编码过程，因此一个 \(s\) 步因果扩散模型每个分块实际上需要 \(s\) 次函数评估（NFE）。噪声上下文通过复用最后一个去噪步骤的 KV 状态作为后续分块的上下文，消除了这一额外过程，将有效延迟从 \(s\) 次 NFE 降低到 \(1\) 次 NFE。除了加速之外，噪声上下文还能提升长程鲁棒性，因为残余噪声充当了低通滤波器，在抑制累积的高频伪影的同时保留了粗粒度的运动动态（Huang 等人，2025b）。
在 TF 阶段，噪声上下文通过将打包的 TF 前向传播中的干净历史替换为对应上下文时间步的含噪历史 token 来实现，而损失函数仍然仅应用于当前目标块。在 SF 阶段，噪声上下文在自回归推理过程中直接使用。虽然在 TF 阶段引入噪声上下文能更好地与推理对齐，但我们发现实践中仅在最终 SF 阶段应用它就足够了。
自定义步数调度
去噪步数也可以在不同分块之间变化。在文生视频生成中，第一个分块通常要求更高，因为它要建立全局场景、布局和外观，而后续分块主要是在先前上下文的基础上扩展视频。因此，我们允许采用依赖于分块的步数调度
| (20) |
其中 \(s_i\) 表示第 \(i\) 个分块的去噪步数。例如，一个标称的 2 步模型可以使用 \(s_1 = 4, s_2 = 2\)，仅将额外计算分配给第一个分块。
对于 SF-DMD 训练，我们根据训练迭代次数循环改变推理长度。例如，对于目标调度 \(S = [4, 2]\)，SF-DMD 会反复循环 \([4, 2]\)。这种循环策略很重要，因为 SF-DMD 仅通过每个分块的最后一个去噪步骤进行反向传播。循环推理长度使得不同的去噪区间在不同迭代轮次中成为最后一个可微步骤，而不是仅监督最大步数采样器的最后一个区间。
3.2 基础设施
| 代码库 | 方案范围 | 算法方案 | 双向基础设施 | 因果基础设施 | |||||||||||
| 双向 | 因果 | TF | DF | SF | 重放 | FSDP2 | CP/SP | SAC | JVP | FSDP2 | CP/SP | SAC | JVP | KV 缓存 | |
| Self-Forcing (Huang 等人, 2025a) | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | v1 | ✗ | AC | ✗ | ✓post |
| FastVideo (Hao-AI Lab, 2026) | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓F-U | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓post |
| FastGen (Nie 等人, 2026a) | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | AC | ✗ | ✓ | ✗ | AC | ✗ | ✓post |
| (Causal-)rCM | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓F-U | ✓ | ✓ | ✓ | ✓F-U | ✓ | ✓ | ✓pre/post |
注：✓：支持；✗：未发现；：部分支持、不明确或路径依赖。
TF：教师强制，实现为[干净帧，噪声帧]拼接并带有特殊因果掩码。
DF：扩散强制，使用普通块因果掩码。
SF：自强制，采用自展开 / KV 缓存式训练执行。
重放：重放反向传播技术，避免在自展开过程中存储整个计算图。
FSDP2：全分片数据并行 v2。v1：仅支持 FSDP1。
CP/SP：上下文/序列并行。T：时间/帧轴；F：展平的视频 token 轴，例如展平的 THW 块 token。U：DeepSpeed-Ulysses；R：环注意力；UR：Ulysses–环混合（USP）。
SAC：选择性激活检查点。AC：激活检查点，但未明确为操作级别的 SAC。
JVP：雅可比向量积。连续时间一致性模型（sCM/MeanFlow）的基础算子。
KV 缓存：因果自注意力 KV 缓存。pre：在 RoPE 之前缓存 K；post：在 RoPE 之后缓存 K。
Causal-rCM 被设计为一种算法-基础设施方案。其主要基础设施目标是使因果训练范式（TF、DF 和 SF）、基于连续时间 JVP 的 CM 以及大规模并行训练能够相互兼容。实现这一目标需要精心协同设计注意力掩码规范、KV 缓存、FSDP2、上下文并行、激活检查点、FlashAttention-2 JVP 内核以及重放反向传播。表 2 总结了由此产生的系统级覆盖范围，并突出了 Causal-rCM 相对于其他广泛使用的代码库在基础设施方面的优势。
3.2.1 主要组件
带自定义掩码的 FlashAttention-2 JVP 内核
连续时间一致性模型需要网络输出沿教师常微分方程（ODE）的切线。对于大型视频Transformer而言，使用通用的`torch.func.jvp`在非融合注意力上计算该切线是不切实际的，因为这会实例化大型注意力中间结果并带来相应的内存开销。为了在TF掩码下实现通过融合注意力的JVP计算，我们基于rCM（Zheng等人，2025d）中的FlashAttention2-JVP内核进行了构建，并将其扩展以支持自定义掩码。TF掩码被表示为可接受的查询-键范围，而非实例化的稠密矩阵。具体细节见附录B。
并行化
我们使用FSDP2（Zhao等人，2023）作为ZeRO-3风格的分片后端：参数、梯度和优化器状态在数据并行秩之间进行分区，每个模块仅为其本地计算实例化完整参数。这减少了每块GPU的模型状态内存，使得在同一蒸馏流程中训练包含学生网络、教师网络、伪分数网络和EMA网络的大型视频DiT模型成为可能。我们使用分布式检查点（DCP）来跨秩直接保存和恢复分片后的模型及优化器状态，避免了在单个进程上收集完整模型状态的需求。
我们采用扁平化的Ulysses风格上下文并行（CP）（Jacobs等人，2023）来将长视频token序列跨秩进行分片。具体来说，时空视频token首先被展平为单个序列，然后CP将此展平后的序列维度在P个设备上进行分区。在注意力计算之前，每块GPU持有QKV的一个大小为[B, H, L/P, C]的分片。随后通过一次all-to-all操作将QKV重新分布为[B, H/P, L, C]以进行本地注意力计算，之后再通过另一次all-to-all操作恢复注意力输出的序列分区。一个关键的设计选择是使CP对外层算法透明：网络接口始终接收和返回全局完整序列，与CP大小无关，而网络内部则处理本地序列分片、all-to-all注意力计算以及输出收集。
激活检查点
我们采用选择性激活检查点（SAC）来减少激活内存，其原理是在反向传播过程中仅重计算网络中的选定部分。与基于区域的普通 `torch.utils.checkpoint` 不同，SAC 提供了更细粒度的控制，能够决定哪些操作需要重计算、哪些中间结果需要保留。在实际应用中，我们主要将 SAC 应用于计算密集的无状态区域，例如注意力模块和 MLP 模块，而将轻量级或有状态的操作保留在检查点区域之外。
KV 缓存
KV 缓存用于因果式逐步执行和推理。我们区分了三种缓存模式：禁用模式用于普通的打包训练，追加模式用于将已生成的块提交到缓存中，只读模式用于在生成当前块的同时关注先前已提交的块。缓存的 K/V 张量在构造时即被分离，这阻止了梯度通过先前块传播，并使 SF-DMD 的内存保持有界。缓存还记录了块的边界，因此只读前向传播可以仅暴露当前块所需的前缀。这既支持标准的自回归逐步展开，也支持诸如噪声上下文等变体，其中最终的去噪前向传播可以复用为上下文状态。
我们同时支持预 RoPE 和后 RoPE 的键缓存。后 RoPE 缓存简单高效，因为缓存的键可以直接复用。预 RoPE 缓存在相同缓存内容可能需要不同位置处理时非常有用，例如用于长度外推或替代位置索引（Yesiltepe 等人，2026；Yi 等人，2025；Li 等人，2026b；Kim 等人，2026）。该实现将此选择保留在注意力上下文中，因此高层级的逐步展开代码无需区分这两种情况。
重放反向传播
SF-DMD 通过自回归 rollout 生成 on-policy 视频。在采用梯度截断的标准执行流程中，所有分块的最后可微去噪步骤会保留在计算图中，这对于长视频而言可能消耗大量内存。因此，我们提供了一种可选的回放反向传播模式（Hong 等人，2025）作为节省内存的实现方案。Rollout 首先在不计算梯度的情况下构建，同时存储每个分块最终的噪声输入、时间步、分离的 KV 缓存以及 DMD 目标。随后，每个分块的最后去噪步骤在启用梯度的情况下重新计算，其梯度通过梯度累积单独反向传播。这种方式以额外的计算量换取更低的激活内存占用。我们特意为 SF-DMD 保留了这条回放路径：TF、DF 和 TF-CM 仍保持打包状态，因为在启用 SAC 后，回放可微前缀 KV 计算带来的额外收益有限。
3.2.2 兼容性设计
Causal-rCM 的一个主要目标是使高级因果训练特性具备可组合性。在实践中，许多独立工作的组件在同时使用时可能会产生冲突。因此，我们在执行语义层面实现兼容性，而非将其作为独立的特性开关。
SAC FlexAttention。
打包的 TF/DF/TF-CM 训练依赖于自定义掩码注意力机制。在 FlexAttention 路径中，注意力模式由 mask_mod 函数指定，并由 PyTorch 编译器降级为专用的融合注意力内核。为了使其与 SAC 兼容，我们使用 torch>=2.10 并结合
| torch._inductor.config.wrap_inductor_compiled_regions = True |
该设置将 Inductor 编译后的 FlexAttention 调用作为显式的可检查点区域暴露给 SAC，在内部表示为 inductor_compiled_code。
SAC 自强制机制。
SF-DMD 的 rollout 是有状态的，因为 KV 缓存和因果元数据会跨数据块演变。我们通过将持久化缓存存储与每次前向传播的因果状态分离，使其与 SAC 兼容：历史 K/V 张量作为分离的上下文存储，而每次前向传播构建一个全新的 CausalInferenceState，描述当前数据块、缓存范围以及用于未来重计算的追加/只读模式。推理状态不会通过原地更新被复用，因此检查点重计算会重建与原始前向传播相同的因果上下文。缓存追加前向传播被排除在检查点执行之外，因此重计算永远不会重放缓存变更；检查点区域仅读取固定的因果上下文。
JVP FSDP2。
遵循 rCM，我们在层级别实现 JVP，而不是对 FSDP2 包装的模型应用全局的 torch.func.jvp。每一层暴露一个配对的 primal-tangent 接口，接收并返回 。这对应于 FSDP2(JVP) 设计，而非 JVP(FSDP2)。FSDP2 继续在层边界管理参数物化、分片和梯度归约，而 tangent 传播则在每一层的前向计算内部本地执行。
JVP Ulysses CP。
Ulysses CP 自然地扩展到 JVP，因为 tangent 张量遵循与其 primal 对应张量相同的通信模式。具体来说， 与 一起进行 all-to-all 交换，局部注意力计算被替换为我们自定义掩码的 FlashAttention-2 JVP 内核，生成的 通过相同的输出 all-to-all 返回，如同 。我们复用了 rCM 中与 JVP 兼容的分布式注意力设计，同时为打包的 TF/DF/TF-CM 训练添加了自定义掩码支持。
KV 缓存 Ulysses CP。
对于 rollout 执行，缓存的 K/V 张量必须与 Ulysses CP 兼容。我们使用后 all-to-all KV 缓存，其中缓存以与暴露给局部注意力相同的 [B, H/P, L, C] 布局存储。每个 CP 等级直接复用其头部分片、全序列的缓存 K/V 状态。这避免了在全局布局和 CP 本地布局之间反复转换旧的缓存条目。
4 实验
4.1 设置
| 配置 | 阶段 1 | 阶段 2 | 阶段 3 | ||
| Wan2.1-1.3B TF/DF | Wan2.1-14B TF/DF | Wan2.1-1.3B TF-dCM | Wan2.1-1.3B TF-sCM | Wan2.1-1.3B SF-DMD | |
| 全局批次大小 | 256 | 64 | 32 | 32 | 64 |
| 上下文并行大小 | 1 | 8 | 4 | 4 | 4 |
| 学生优化器 | AdamW lr wd | AdamW lr wd | AdamW lr wd | AdamW lr wd | AdamW lr wd |
| 伪分数优化器 | – | – | – | – | AdamW lr wd |
| CFG 缩放系数 | – | – | 3.0 | 3.0 | 5.0 |
| 时间采样/加权 | TF: , 共享 , 高斯钟形权重; DF: , 随机逐块 , 无权重 | TF: , 共享 , 高斯钟形权重; DF: , 随机逐块 , 无权重 | 均匀 RF 网格，带偏移 , 步数 , 跳过 | ||
| 特定超参数 | – | – | – | 切线预热 | 最大展开步数 学生更新频率 |
| 训练迭代次数 | 30k | 30k | 10k | 1k | 变化 |
模型与数据集。
我们在 Wan2.1 T2V（Wan 等人，2025）上以 480p 分辨率进行主要的流式视频生成实验。视频以空间分辨率生成，包含 81 个 RGB 帧，对应 VAE 时间压缩后的 21 个潜在帧。训练使用 rCM（Zheng 等人，2025d）提供的合成 T2V 数据，该数据由双向 Wan2.1-14B 教师模型通过 100 步欧拉采样、偏移 3.0 和 CFG 缩放系数 5.0 生成。我们使用 Wan2.1-1.3B 作为主要学生模型，并使用 Wan2.1-14B 教师模型进行蒸馏。
我们评估两种因果块模式。帧级设置（记为 c1-1）使用一个初始潜在帧，然后使用单潜在帧的块。块级设置（记为 c3-3）使用一个包含三帧的初始潜在块，然后使用三潜在帧的块。相同的块模式一致地用于打包的 TF/DF/TF-CM 掩码、SF-DMD 展开、KV 缓存推理和流式评估。
训练。
Causal-rCM 采用三阶段训练方案。我们在表 3 中报告主要超参数。对于 TF-CM，我们使用经过 TF 训练的 14B 因果教师模型。对于 SF-DMD，我们使用 14B 双向教师模型和伪分数网络。
对于少步 SF-DMD，我们使用最多 4 个去噪步的 RF 采样调度。4 步采样器使用中间时间 。2 步采样器对第一个块使用 4 步，对后续块使用 2 步，调度为 。2 步噪声上下文变体使用调度 ，并重用最终去噪前向传播作为上下文缓存。1 步变体对第一个块使用 4 步，对后续块使用 1 步，调度为 。
评估指标。
对于流式质量，我们使用 VBench-T2V（Huang 等人，2024）评估文生视频任务，报告总分以及质量和语义子分数。
对于推理效率，我们报告函数评估次数（NFE）、以每秒帧数（FPS）为单位的吞吐量、首个数据块延迟和第二个数据块延迟。所有效率测量均在单张 H100 GPU 上以批次大小 1 进行。报告的 FPS 和延迟包括扩散采样和 VAE 解码。
4.2 结果
4.2.1 流式视频生成
主要结果。
表 4 将 Causal-rCM 与双向 Wan2.1 以及流式视频生成基线进行了比较，包括 Self-Forcing（Huang 等人，2025a）、LongLive（Yang 等人，2026）、Causal Forcing（Zhu 等人，2026）和 AnyFlow（Gu 等人，2026）。我们报告了逐帧和逐数据块的结果。Causal-rCM 在支持 4 步、2 步、2 步噪声上下文和 1 步推理调度的情况下，实现了最先进的流式质量。
| 方法 | NFE | 总分 | 质量分数 | 语义分数 | 吞吐量 | 首个延迟 | 第二个延迟 | SF-DMD 迭代次数 |
| (FPS) | (秒) | (秒) | ||||||
| 双向 | ||||||||
| Wan2.1-1.3B | 502 | 82.78 | 83.44 | 80.13 | 0.72 | – | – | – |
| Wan2.1-14B | 502 | 83.35 | 83.97 | 80.88 | 0.18 | – | – | – |
| 逐帧 (c1-1) | ||||||||
| Causal Forcing (4 步) | 5 | 81.56 | 82.59 | 77.44 | 8.3 | 0.40 | 0.46 | – |
| Causal-rCM (4 步) | 5 | 84.29 | 85.27 | 80.36 | 8.3 | 0.40 | 0.46 | 1200 |
| Causal-rCM (2 步) | 3 | 84.63 | 85.46 | 81.31 | 12.2 | 0.40 | 0.31 | 3000 |
| Causal-rCM (2 步，噪声上下文) | 2 | 83.11 | 83.55 | 81.37 | 15.9 | 0.40 | 0.23 | 1500 |
| Causal-rCM (1 步) | 2 | 84.63 | 85.54 | 81.01 | 15.9 | 0.40 | 0.23 | 3000 |
| 逐数据块 (c3-3) | ||||||||
| Self-Forcing (4 步) | 5 | 83.76 | 84.53 | 80.68 | 17.4 | 0.57 | 0.64 | – |
| LongLive (4 步) | 5 | 83.62 | 84.36 | 80.69 | 17.4 | 0.57 | 0.64 | – |
| Causal Forcing (4 步) | 5 | 83.96 | 84.94 | 80.04 | 17.4 | 0.57 | 0.64 | – |
| AnyFlow (4 步) | 5 | 84.31 | 85.15 | 80.94 | 17.4 | 0.57 | 0.64 | – |
| Causal-rCM (4 步) | 5 | 84.37 | 85.02 | 81.73 | 17.4 | 0.57 | 0.64 | 1250 |
| Causal-rCM (2 步) | 3 | 84.30 | 85.04 | 81.36 | 22.2 | 0.57 | 0.49 | 2500 |
| Causal-rCM (2 步，噪声上下文) | 2 | 84.24 | 84.96 | 81.36 | 25.6 | 0.57 | 0.41 | 1750 |
| Causal-rCM (1 步) | 2 | 84.01 | 84.71 | 81.22 | 25.6 | 0.57 | 0.41 | 3000 |
在自定义步进调度和噪声上下文下的性能。
表 4 展示了在自定义步长调度下的一种有趣行为。在逐帧设定中，1 步和 2 步的 Causal-rCM 模型表现优于 4 步变体，这乍看之下有违直觉。我们将此归因于逐帧设定的本质：每个自回归块仅包含单个潜在帧，因此没有需要去噪的内部时间结构。在这种情况下，为每个未来块分配大量去噪步长会过度放大自回归反馈误差，尤其是考虑到 SF-DMD 的梯度截断策略。根据经验，我们观察到 4 步逐帧 SF-DMD 更容易出现相机漂移，例如样本间一致的左向相机旋转，并且只能稳定训练约 1k 次迭代。相比之下，对后续块使用 1 或 2 步能大幅抑制这种漂移，并允许稳定训练约 3k 次迭代。由于每个未来块仅包含一个潜在帧，1–2 步去噪已足以生成该帧，而减少的展开深度则提升了稳定性。
在逐块设定中趋势有所不同，其中每个块包含三个潜在帧，因此具有不可忽视的内部时间相关性。在此情况下，更深的 4 步采样器能为建模运动与块内一致性提供更好的去噪轨迹，从而取得最佳总体得分。这表明最优步长调度取决于每个自回归块的时间跨度：逐帧生成更受益于浅层、稳定的展开，而逐块生成则受益于更深的去噪深度。
噪声上下文通过消除额外的干净上下文 KV 编码传递，进一步提高了推理效率，将每块的有效成本降低至 NFEs。比较带噪声上下文的 2 步采样与 1 步采样，我们发现 1 步采样在逐帧设定下表现更好，而带噪声上下文的 2 步采样在逐块设定下表现更好。这与上述观察结果一致。对于单帧块，额外的去噪步骤带来的收益有限，而上下文中的残余噪声会直接影响帧级预测中的细粒度细节。对于三帧块，该块包含一个维度更高、冗余更多的时空 token 组。在这种模式下，高斯扰动不太可能均匀地破坏整个块级结构，并且大部分运动与粗粒度语义上下文仍可保留（Hoogeboom 等人，2023）。因此，带噪声上下文的 2 步采样能够保留额外去噪步骤对块内时间连贯性的益处。
TF-dCM 与 TF-sCM 之间的比较。
图 6 比较了最终 SF-DMD 阶段之前的 TF-dCM 与 TF-sCM。TF-sCM 始终能以更少的训练迭代次数提供更强的初始化。在逐帧设定下，TF-sCM 在 1-2k 次迭代内达到 81.8 以上的 VBench-T2V 分数，已经超越了训练了 10k 次迭代的 TF-dCM。在逐块设定下，差距更为明显：TF-sCM 在 1-2k 次迭代内达到 83 以上，而 TF-dCM 改进速度更慢，并且在更长时间的训练后仍保持在较低水平。
初始化策略的消融研究。
| 初始化 | 总分 | 质量分 | 语义分 | SF-DMD 迭代次数 |
| 逐帧（c1-1） | ||||
| DF | 83.11 | 83.85 | 80.16 | 2000 |
| TF | 82.62 | 83.62 | 78.61 | 1000 |
| DF-KD | 80.59 | 80.41 | 81.32 | 2000 |
| TF-KD | 83.49 | 84.50 | 79.43 | 1250 |
| TF-dCM | 84.29 | 85.27 | 80.36 | 1200 |
| TF-sCM | 83.84 | 84.67 | 80.55 | 1000 |
| 逐块（c3-3） | ||||
| DF | 84.80 | 85.58 | 81.65 | 1500 |
| TF | 84.95 | 85.82 | 81.47 | 1000 |
| DF-KD | 83.61 | 84.10 | 81.68 | 1500 |
| TF-KD | 83.79 | 84.41 | 81.30 | 1000 |
| TF-dCM | 84.33 | 85.22 | 80.75 | 3200 |
| TF-sCM | 84.37 | 85.02 | 81.73 | 1250 |
表 5 对 SF-DMD 的初始化策略进行了消融实验。我们比较了来自 DF 和 TF 的因果扩散初始化、ODE 对知识蒸馏变体（DF-KD 和 TF-KD），以及教师强制一致性初始化（TF-dCM 和 TF-sCM）。相应的训练曲线如图 7 所示。
在逐帧设定下，TF-CM 初始化取得了最佳整体性能，DF 和 TF-KD 也提供了有竞争力的替代方案。虽然 TF-sCM 从更强的初始模型出发，但 TF-dCM 在 SF-DMD 过程中更稳定，支持更长的精炼过程，从而获得了更高的峰值分数。在逐块设定下，DF/TF 初始化取得了最高的 VBench-T2V 分数，接近 85。然而，如图 8 所示，这些模型生成的纹理（如水、头发和树叶）往往过度平滑和过饱和，精细细节明显更少。综合考虑 VBench 分数和定性检查，TF-CM 初始化仍然是最可靠的选择。在两种 TF-CM 变体中，TF-sCM 略优于 TF-dCM，同时所需的 SF-DMD 迭代次数更少。
4.2.2 交互式世界模型
我们进一步将 Causal-rCM 应用于 Cosmos 3（NVIDIA，2026），这是一个基于双塔混合 Transformer 架构的全模态世界模型。Cosmos 3 将用于文本和提示词推理的理解塔（UND）与用于视觉、动作和声音 token 的生成塔（GEN）分离开来，同时跨模态共享多模态注意力层和统一的 3D mRoPE。在原始生成器模式下，GEN token 使用双向自注意力进行多模态去噪。为了支持交互式世界建模，我们将 GEN 视觉流转换为时间因果自回归扩散堆栈（图 9）。
我们将每个潜在视频帧视为一个视觉超 token，它包含该帧的所有空间潜在 token。时间因果注意力在超 token 级别应用：未来的视觉超 token 对过去和当前的超 token 进行掩码，而同一视觉超 token 内的空间 token 则保持完全双向。
相同的因果堆栈支持文本到视频、图像到视频以及前向动力学（动作条件）建模。在文本到视频中，所有视觉超 token 都从文本条件生成。在图像到视频和前向动力学中，第一个视觉超 token 作为干净上下文提供，模型自回归地预测未来的视觉超 token。对于前向动力学，动作超 token 作为输入条件处理。第一帧使用空动作超 token，真实动作超 token 通过统一的 3D mRoPE 与下一个生成的视觉超 token 对齐，从而使动作控制从状态到状态的转换。
如图 10 所示，交互式 Cosmos 3 模型支持流式控制：给定相同的初始场景，生成的未来帧在左转、右转和直行控制下遵循不同的轨迹。
5 相关工作
生成建模中的微分信息与 JVP。
差分信息在扩散常微分方程中扮演了重要角色，其作用超越了标准的一阶去噪监督。高阶去噪分数匹配表明，对于最大似然扩散常微分方程训练而言，一阶分数匹配是不够的，它通过控制高阶分数误差来缩小似然差距（Lu 等人，2022）。后续工作通过速度参数化、方差缩减和高阶流匹配目标，改进了扩散常微分方程的似然估计与训练（Zheng 等人，2023b）。DPM-Solver-v3 进一步利用预训练扩散模型的经验模型统计量，推导出改进的常微分方程求解器系数，并揭示了扩散网络中与时间导数相关的数值问题（Zheng 等人，2023a）。最近，sCM、MeanFlow、AYF 和 FACM 将 JVP 作为连续时间一致性或流映射目标的直接训练信号（Lu 和 Song，2024；Geng 等人，2025；Sabour 等人，2025；Peng 等人，2025）。rCM 通过使 JVP 计算兼容 FlashAttention、FSDP 和上下文并行，将基于 JVP 的一致性蒸馏扩展到大型图像和视频扩散模型，并将其与 DMD 正则化相结合（Zheng 等人，2025d）。Causal-rCM 将这一方向扩展到自回归视频扩散，在干净的因果上下文中应用基于 JVP 的教师强制 sCM，作为自强制 DMD 的结构化初始化。
蒸馏目标中的前向-反向互补性。
越来越多的少步方法可被视为将保持覆盖的前向组件与追求质量或奖励的反向组件相结合。在文生图领域，近期的实践研究标准化了针对强文本条件教师模型的大规模少步知识蒸馏方案，并实证比较了 sCM 与 MeanFlow（Pu 等人，2025）。流映射方法更直接地蒸馏教师 ODE 行为。FreeFlow（Tong 等人，2025）通过从先验分布采样并在学生模型诱导的流映射状态上查询教师模型动态，结合额外的校正目标以减轻累积误差，实现了无数据的流映射蒸馏。相比之下，-Flow（Chen 等人，2025b）更明确地采用在策略方式，沿学生策略自身的 ODE 轨迹匹配教师模型的速度。分布匹配方法提升了少步生成质量，但可能牺牲多样性；因此，近期变体引入了角色分离、强化学习信号或对抗流目标，以平衡模式覆盖与模式追求（Jiang 等人，2025；Wu 等人，2026；Cheng 等人，2025；Lin 等人，2026）。这种互补性在近期的长视频工作中尤为明显：Cai 等人（2026）将用于长程结构的监督全局流匹配头与用于短窗口保真度的局部 DMD 头配对，而 HiAR（Zou 等人，2026）观察到自展开反向 KL 蒸馏可能放大低运动捷径，并添加了前向 KL 正则化项以保持运动多样性。
视频与自回归扩散蒸馏。
视频蒸馏除了要处理逐帧视觉质量外，还必须处理时间一致性和长程误差累积问题。Self-Forcing（Huang 等人，2025a）和 APT2（Lin 等人，2025b）是具有代表性的工作，它们提出将 self-forcing 作为一种在线策略蒸馏范式，用于缓解自回归生成中的曝光偏差。具体而言，APT2 使用教师强制一致性蒸馏来初始化 self-forcing，但在 self-forcing 阶段依赖于相对繁琐的 GAN 目标函数。与我们工作同期，Causal Forcing++（Zhao 等人，2026）也将教师强制一致性与 self-forcing DMD 相结合，而我们则在教师强制下实现了基于 JVP 的连续时间一致性，并提供了一个包含系统算法与基础设施的开放配方，以及全面的评估。除了 CM 路线之外，Transition Matching Distillation 使用条件流头将多步视频去噪轨迹与少步过渡过程进行匹配，随后在流头展开上进行分布匹配（Nie 等人，2026b）。AnyFlow 将视频蒸馏从端点一致性转向任意间隔流映射转换，并在双向和因果架构中使用反向模拟进行在线策略蒸馏（Gu 等人，2026）。其他近期工作研究了使用高效解流目标从头开始进行少步视频训练（Park 等人，2026），以及针对过饱和和时间坍塌的视频专用蒸馏损失（You 等人，2026）。对抗性精炼也被探索用于一步式自回归视频生成，例如，通过使用带噪潜变量 GAN 损失来增强 DMD（Feng 等人，2026），或在分布匹配预热后使用非对称对抗性蒸馏（Li 等人，2026a）。蒸馏后的模型与注意力层面的加速是正交的，可以进一步与稀疏注意力技术结合（Zhang 等人，2025a，2026），正如 TurboDiffusion（Zhang 等人，2025b）所展示的那样，它将 rCM 与注意力加速和量化相结合。
6 局限性与未来方向
局限性。
尽管 Causal-rCM 为自回归扩散蒸馏提供了一套有效的算法-基础设施方案，但仍存在若干局限性。首先，采用长展开深度的逐帧 T2V 训练仍然不稳定。在此设置下，4 步 SF-DMD 模型在长时间训练后容易出现相机漂移，例如产生一致的定向相机偏差，因此无法进行长时间训练。这个问题在动作条件化的交互式设置中可能被消除，因为动作提供了显式的运动先验，减少了相机演化的模糊性。其次，SF-DMD 之前的最佳初始化并不总能转化为最佳最终模型。TF-sCM 比 TF-dCM 提供了更强的预 SF-DMD 初始化，但在逐帧设置下，TF-dCM 在长时间 SF-DMD 精炼过程中可能更稳定，并达到更高的最终峰值。这表明初始化质量与精炼稳定性并非完全一致。第三，像 rCM 这样的完全联合优化仍然具有挑战性。在我们的因果设置中，联合训练往往会降低 VBench 的上限，因此我们目前采用分阶段流水线。这可能是由于因果教师模型与双向教师模型之间的分布差异造成的。最后，当前的自定义掩码 FlashAttention JVP 内核是用 Triton 实现的。因此，TF-sCM 的每轮迭代速度仅与使用标准 FlashAttention-2 的 TF-dCM 相当，落后于 FlashAttention-3/4 等更先进的内核。
未来方向。
一个自然的下一步是使分阶段方案更加系统化。表 6 总结了我们的高层观点：当前的蒸馏方法可以被解释为两条最终流水线的子集，即 CM 路线和 CTM 路线，每条路线都有双向和因果变体。离散时间方法（dCM、带有限差分估计的 MeanFlow）可以作为连续时间 JVP 方法（sCM、MeanFlow）的预热阶段，以增强稳定性。
| 路线 | 设置 | 最终流水线 | 相关工作 |
| CM | 双向 | dCM sCM DMD/GAN CM/on-policy CM | APT (Lin et al., 2025a): dCM GAN; rCM (Zheng et al., 2025d): sCM DMD。 |
| CM | 因果 | TF-dCM TF-sCM SF-DMD/SF-GAN TF-CM/SF-CM | APT2（Lin 等人，2025b）：TF-dCM SF-GAN；CF++（Zhao 等人，2026）：TF-dCM SF-DMD；Causal-rCM（我们的方法）：TF-dCM/TF-sCM SF-DMD。 |
| CTM | 双向 | MeanFlow（有限差分）MeanFlow（JVP）DMD/GAN MeanFlow/在策略 MeanFlow | Transition Matching（Nie 等人，2026b）：MeanFlow（有限差分）带流头展开的 DMD2-v；AnyFlow（Gu 等人，2026）：MeanFlow（有限差分）DMD + 在策略 MeanFlow（有限差分）。 |
| CTM | 因果 | TF-MeanFlow（有限差分）TF-MeanFlow（JVP）SF-DMD/SF-GAN TF-MeanFlow/SF-MeanFlow | AnyFlow（Gu 等人，2026）：TF-MeanFlow（有限差分）SF-DMD + SF-MeanFlow（有限差分）。 |
除了算法设计之外，未来的工作还应改进底层系统栈。为自定义注意力机制、JVP 和 KV 缓存执行提供更好的内核，结合 torch.compile、CUDA Graphs 和 NVFP4 等运行时特性，可以进一步降低开销，使大规模训练和推理更加高效。
参考文献
- Ali 等人（2025）Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, 等。面向物理 AI 的基于视频基础模型的世界模拟。arXiv 预印本 arXiv:2511.00062，2025 年。
- Arriola 等人（2025）Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, 和 Volodymyr Kuleshov。块扩散：在自回归与扩散语言模型之间插值。arXiv 预印本 arXiv:2503.09573，2025 年。
- Bao 等人（2024）Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, 和 Jun Zhu。Vidu：一种高度一致、动态且富有技巧的基于扩散模型的文本到视频生成器。arXiv 预印本 arXiv:2405.04233，2024 年。
- Brooks 等人（2024）Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, 等。作为世界模拟器的视频生成模型。2024 年。URL https://openai. com/research/video-generation-models-as-world-simulators，3，2024 年。
- Cai 等人 (2026) Shengqu Cai, Weili Nie, Chao Liu, Julius Berner, Lvmin Zhang, Nanye Ma, Hansheng Chen, Maneesh Agrawala, Leonidas Guibas, Gordon Wetzstein 等。模式寻求与均值寻求相结合，实现快速长视频生成。arXiv 预印本 arXiv:2602.24289，2026 年。
- Chen 等人 (2024) Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake 和 Vincent Sitzmann。扩散强制：下一 token 预测与全序列扩散相结合。神经信息处理系统进展，37:24081–24125，2024 年。
- Chen 等人 (2025a) Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Junchen Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengcheng Ma 等。Skyreels-v2：无限长度电影生成模型。arXiv 预印本 arXiv:2504.13074，2025a。
- Chen 等人 (2025b) Hansheng Chen, Kai Zhang, Hao Tan, Leonidas Guibas, Gordon Wetzstein 和 Sai Bi。pi-flow：基于策略的少步生成，通过模仿蒸馏实现。arXiv 预印本 arXiv:2510.14974，2025b。
- Chen 等人 (2026) Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, Ying-Cong Chen, Enze Xie, Xiaojuan Qi 和 Song Han。Longlive2.0：用于长视频生成的 NVFP4 并行基础设施。arXiv 预印本 arXiv，2026 年。
- Cheng 等人 (2025) Zhenglin Cheng, Peng Sun, Jianguo Li 和 Tao Lin。Twinflow：利用自对抗流在大模型上实现单步生成。arXiv 预印本 arXiv:2512.05150，2025 年。
- Dong 等人 (2024) Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang 和 Horace He。Flex Attention：一种用于生成优化注意力内核的编程模型。arXiv 预印本 arXiv:2412.05496，2(3):4，2024 年。
- Feng 等人 (2026) Jiaqi Feng, Justin Cui, Yuanhao Ban 和 Cho-Jui Hsieh。One-forcing：迈向稳定的单步自回归视频生成。arXiv 预印本 arXiv:2605.23458，2026 年。
- Feng 等人 (2025) Yao Feng, Chendong Xiang, Xinyi Mao, Hengkai Tan, Zuyue Zhang, Shuhe Huang, Kaiwen Zheng, Haitian Liu, Hang Su 和 Jun Zhu。Vidarc：用于闭环控制的具体化视频扩散模型。arXiv 预印本 arXiv:2512.17661，2025 年。
- Gao 等人 (2025) Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li 等。Seedance 1.0：探索视频生成模型的边界。arXiv 预印本 arXiv:2506.09113，2025 年。
- Geng 等人 (2025) Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter 和 Kaiming He。一步生成建模的均值流。arXiv 预印本 arXiv:2505.13447，2025 年。
- Gu 等人 (2026) Yuchao Gu, Guian Fang, Yuxin Jiang, Weijia Mao, Song Han, Han Cai 和 Mike Zheng Shou。Anyflow：基于在线策略流图蒸馏的任意步视频扩散模型。arXiv 预印本 arXiv:2605.13724，2026 年。
- Hao-AI Lab (2026) Hao-AI Lab。FastVideo：用于加速视频生成的统一推理与后训练框架，2026 年。网址 https://github.com/hao-ai-lab/FastVideo。
- He 等人 (2025) Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Size Wu, Wei Li, Xuchen Song, Yang Liu, Yangguang Li 和 Yahui Zhou。Matrix-game 2.0：一个开源的实时流式交互世界模型。arXiv 预印本 arXiv:2508.13009，2025 年。
- Ho 等人 (2020) Jonathan Ho, Ajay Jain 和 Pieter Abbeel。去噪扩散概率模型。神经信息处理系统进展，33:6840–6851，2020 年。
- Hong 等人 (2025) Yicong Hong, Yiqun Mei, Chongjian Ge, Yiran Xu, Yang Zhou, Sai Bi, Yannick Hold-Geoffroy, Mike Roberts, Matthew Fisher, Eli Shechtman, Kalyan Sunkavalli, Feng Liu, Zhengqi Li 和 Hao Tan。Relic：具有长时记忆的交互式视频世界模型。arXiv 预印本 arXiv:2512.04040，2025 年。
- Hoogeboom 等人 (2023) Emiel Hoogeboom, Jonathan Heek 和 Tim Salimans。简单扩散：高分辨率图像的端到端扩散。国际机器学习大会论文集，第 13213–13232 页。PMLR，2023 年。
- Huang 等人 (2025a) Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou 和 Eli Shechtman。自强迫：弥合自回归视频扩散中的训练-测试差距。arXiv 预印本 arXiv:2506.08009，2025a。
- Huang 等人 (2025b) Yubo Huang, Hailong Guo, Fangtai Wu, Weiqiang Wang, Shifeng Zhang, Shijie Huang, Qijun Gan, Lin Liu, Sirui Zhao, Enhong Chen, Jiaming Liu, 和 Steven Hoi。Live avatar：基于音频驱动的流式实时无限长度虚拟形象生成。arXiv 预印本 arXiv:2512.04677，2025b。
- Huang 等人 (2024) Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit 等人。Vbench：视频生成模型的综合基准测试套件。收录于 IEEE/CVF 计算机视觉与模式识别会议论文集，第 21807–21818 页，2024 年。
- HunyuanWorld (2025) Team HunyuanWorld。Hy-world 1.5：一种具备实时延迟和几何一致性的交互式世界建模系统框架。arXiv 预印本，2025 年。
- Jacobs 等人 (2023) Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, 和 Yuxiong He。Deepspeed ulysses：用于实现超长序列 Transformer 模型训练的系统优化。arXiv 预印本 arXiv:2309.14509，2023 年。
- Jiang 等人 (2025) Dengyang Jiang, Dongyang Liu, Zanyi Wang, Qilong Wu, Liuzhuozheng Li, Hengzhuang Li, Xin Jin, David Liu, Changsheng Lu, Zhen Li 等人。分布匹配蒸馏遇上强化学习。arXiv 预印本 arXiv:2511.13649，2025 年。
- Jin 等人 (2025) Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, 和 Zhouchen Lin。用于高效视频生成建模的金字塔流匹配。收录于国际学习表征会议，第 2025 卷，第 23378–23402 页，2025 年。
- Kim 等人 (2023) Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, 和 Stefano Ermon。一致性轨迹模型：学习扩散的概率流 ODE 轨迹。arXiv 预印本 arXiv:2310.02279，2023 年。
- Kim 等人 (2026) Youngrae Kim, Qixin Hu, C-C Jay Kuo, 和 Peter A Beerel。Memrope：通过演化记忆 token 实现免训练的无限视频生成。arXiv 预印本 arXiv:2603.12513，2026 年。
- Kong 等人 (2024) Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, 等. Hunyuanvideo: 大型视频生成模型的系统框架. arXiv 预印本 arXiv:2412.03603, 2024.
- Li 等人 (2026a) Haobo Li, Yanhong Zeng, Yunhong Lu, Jiapeng Zhu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Yujun Shen, 和 Zhipeng Zhang. Aad-1: 面向单步自回归视频生成的非对称对抗性知识蒸馏. arXiv 预印本 arXiv:2606.03972, 2026a.
- Li 等人 (2026b) Haodong Li, Shaoteng Liu, Zhe Lin, 和 Manmohan Chandraker. Rolling sink: 弥合自回归视频扩散中有限视野训练与开放式测试之间的鸿沟. arXiv 预印本 arXiv:2602.07775, 2026b.
- Li 等人 (2026c) Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, 等. 面向机器人控制的因果世界建模. arXiv 预印本 arXiv:2601.21998, 2026c.
- Lin 等人 (2025a) Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, 和 Lu Jiang. 面向单步视频生成的扩散对抗后训练. 见第42届国际机器学习大会论文集, 机器学习研究论文集第267卷, 页码 37959–37974. PMLR, 2025a.
- Lin 等人 (2025b) Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, 和 Lu Jiang. 面向实时交互式视频生成的自回归对抗后训练. arXiv 预印本 arXiv:2506.09350, 2025b.
- Lin 等人 (2026) Shanchuan Lin, Ceyuan Yang, Zhijie Lin, Hao Chen, 和 Haoqi Fan. 连续对抗流模型. arXiv 预印本 arXiv:2604.11521, 2026.
- Lipman 等人 (2022) Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, 和 Matt Le. 用于生成式建模的流匹配. arXiv 预印本 arXiv:2210.02747, 2022.
- Liu 等人 (2026) Jinxiu Liu, Xuanming Liu, Kangfu Mei, Yandong Wen, Ming-Hsuan Yang, 和 Weiyang Liu. 通过对角蒸馏实现流式自回归视频生成. 见 ICLR, 2026.
- Liu 等人 (2022) Xingchao Liu, Chengyue Gong, 和 Qiang Liu。流式直通与快速：学习通过修正流生成和传输数据。arXiv 预印本 arXiv:2209.03003，2022 年。
- Lu 和 Song (2024) Cheng Lu 和 Yang Song。简化、稳定并扩展连续时间一致性模型。arXiv 预印本 arXiv:2410.11081，2024 年。
- Lu 等人 (2022) Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, 和 Jun Zhu。通过高阶去噪分数匹配对基于分数的扩散 ODE 进行最大似然训练。收录于《国际机器学习大会》，第 14429–14460 页。PMLR，2022 年。
- Luhman 和 Luhman (2021) Eric Luhman 和 Troy Luhman。迭代生成模型中的知识蒸馏以提升采样速度。arXiv 预印本 arXiv:2101.02388，2021 年。
- Nie 等人 (2026a) Weili Nie, Julius Berner, Chao Liu, 和 Arash Vahdat。Nvidia FastGen：扩散模型的快速生成，2026a。URL https://github.com/NVlabs/FastGen。
- Nie 等人 (2026b) Weili Nie, Julius Berner, Nanye Ma, Chao Liu, Saining Xie, 和 Arash Vahdat。面向快速视频生成的过渡匹配蒸馏。arXiv 预印本 arXiv:2601.09881，2026b。
- Ning 等人 (2024) Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, 和 Itir Onal Ertugrul。阐明扩散模型中的曝光偏差。收录于《国际学习表征大会》，第 2024 卷，第 15167–15189 页，2024 年。
- NVIDIA (2026) NVIDIA。Cosmos 3：面向物理 AI 的全模态世界模型。arXiv 预印本 arXiv:2606.02800，2026 年。URL https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf。
- Park 等人 (2026) Dogyun Park, Yanyu Li, Sergey Tulyakov, 和 Anil Kag。Eflow：通过高效解流从零开始快速训练少步视频生成器。arXiv 预印本 arXiv:2603.27086，2026 年。
- Peng 等人 (2025) Yansong Peng, Kai Zhu, Yu Liu, Pingyu Wu, Hebei Li, Xiaoyan Sun, 和 Feng Wu。Facm：流锚定一致性模型。arXiv 预印本 arXiv:2507.03738，2025 年。
- Pu 等人 (2025) Yifan Pu, Yizeng Han, Zhiwei Tang, Jiasheng Tang, Fan Wang, Bohan Zhuang, 和 Gao Huang。面向文生图模型的少步蒸馏：实用指南。arXiv 预印本 arXiv:2512.13006，2025 年。
- Robbyant 团队等 (2026) Robbyant 团队，高泽林，王秋雨，曾艳红，朱嘉鹏，郑家乐，李宜轩，王翰林，徐英豪，马帅磊，陈一航，刘杰，程彦松，姚瑶，朱嘉毅，孟一浩，郑克成，白清岩，陈敬烨，沈泽宏，余悦，朱星，沈宇军，欧阳浩。推进开源世界模型。arXiv 预印本 arXiv:2601.20540，2026 年。
- Sabour 等 (2025) Amirmojtaba Sabour，Sanja Fidler，以及 Karsten Kreis。对齐你的流：扩展连续时间流图蒸馏。arXiv 预印本 arXiv:2506.14603，2025 年。
- Sahoo 等 (2024) Subham Sekhar Sahoo，Marianne Arriola，Yair Schiff，Aaron Gokaslan，Edgar Marroquin，Justin T Chiu，Alexander Rush，以及 Volodymyr Kuleshov。简单有效的掩码扩散语言模型。arXiv 预印本 arXiv:2406.07524，2024 年。
- Schmidt (2019) Florian Schmidt。生成中的泛化：对曝光偏差的深入审视。收录于《第三届神经生成与翻译研讨会论文集》，第 157–167 页，2019 年。
- Seedance 等 (2026) Seedance 团队，陈德，陈立阳，陈鑫，陈颖，陈卓，陈卓伟，程峰，程天恒，程宇峰，等。Seedance 2.0：面向世界复杂性的视频生成技术推进。arXiv 预印本 arXiv:2604.14148，2026 年。
- Shi 等 (2024) 石佳欣，韩可航，王哲，Arnaud Doucet，以及 Michalis K Titsias。面向离散数据的简化与泛化掩码扩散。arXiv 预印本 arXiv:2406.04329，2024 年。
- Song 等 (2020) 宋阳，Jascha Sohl-Dickstein，Diederik P Kingma，Abhishek Kumar，Stefano Ermon，以及 Ben Poole。基于随机微分方程的分数生成建模。arXiv 预印本 arXiv:2011.13456，2020 年。
- Song 等 (2023) 宋阳，Prafulla Dhariwal，Mark Chen，以及 Ilya Sutskever。一致性模型。收录于《国际机器学习大会》，第 32211–32252 页。PMLR，2023 年。
- Teng 等 (2025) 滕汉思，贾宏宇，孙磊，李灵芝，李茂林，唐明秋，韩帅，张天宁，WQ Zhang，罗伟峰，等。Magi-1：大规模自回归视频生成。arXiv 预印本 arXiv:2505.13211，2025 年。
- Tong 等人 (2025) 尚源·童、南野·马、谢赛宁、Tommi Jaakkola。无数据流图蒸馏。arXiv 预印本 arXiv:2511.19428，2025。
- Wan 等人 (2025) Wan 团队、王昂、艾宝乐、文斌、毛超杰、谢晨伟、陈迪、余飞武、赵海明、杨建晓、曾建元、王嘉宇、张景峰、周靖人、王金凯、陈继轩、朱凯、赵康、严可宇、黄良华、冯梦阳、张宁一、李攀登、吴平宇、楚瑞航、冯瑞丽、张世伟、孙思阳、方涛、王天行、桂天一、翁廷宇、沈彤、林伟、王伟、王伟、周文猛、王温特、沈文婷、余文远、石贤忠、黄晓明、徐鑫、寇岩、吕阳宇、李一飞、刘怡静、王益明、张英亚、黄一桐、李勇、吴悠、刘宇、潘玉林、郑云、洪云涛、石宇鹏、冯宇桐、蒋泽因子、韩震、吴志凡、刘子宇。Wan：开放且先进的大规模视频生成模型。arXiv 预印本 arXiv:2503.20314，2025。
- Wang 等人 (2023) 王祯一、陆成、王亦凯、樊宝、李崇轩、苏航、朱军。Prolificdreamer：基于变分分数蒸馏的高保真度与多样化文本到 3D 生成。神经信息处理系统进展，36:8406–8441，2023。
- Wu 等人 (2026) 吴天河、李瑞斌、张磊、马科德。面向快速视觉合成的多样性保持分布匹配蒸馏。arXiv 预印本 arXiv:2602.03139，2026。
- Yang 等人 (2026) 杨帅、黄伟、楚瑞航、肖一程、赵宇阳、王贤邦、李牧阳、谢恩泽、陈颖聪、陆瑶 等。Longlive：实时交互式长视频生成。ICLR，2026。
- Ye 等人 (2025) 叶昊天、郑凯文、徐嘉树、李普恒、陈华宇、韩嘉琪、刘晟、张钦盛、毛汉子、郝泽坤 等。面向扩散模型的大规模数据正则化强化学习。arXiv 预印本 arXiv:2512.04332，2025。
- Ye 等人 (2026) 叶成贤、葛云浩、郑开元、高申远、俞思贤、George Kurian、Suneel Indupuru、You Liang Tan、朱楚宁、项建南 等。世界动作模型是零样本策略。arXiv 预印本 arXiv:2602.15922，2026。
- Yesiltepe 等人（2026 年）Hidir Yesiltepe、Tuna Meral、Adil Kaan Akan、Kaan Oktay 和 Pinar Yanardag。《无限绳：通过自回归自展开实现动作可控的无限视频生成》。载于《IEEE/CVF 计算机视觉与模式识别会议论文集》，第 40256–40265 页，2026 年。
- Yi 等人（2025 年）Jung Yi、Wooseok Jang、Paul Hyunbin Cho、Jisu Nam、Heeji Yoon 和 Seungryong Kim。《深度强制：基于深度汇聚与参与式压缩的无训练长视频生成》。arXiv 预印本，编号 2512.05081，2025 年。
- Yin 等人（2024a 年）Tianwei Yin、Michaël Gharbi、Taesung Park、Richard Zhang、Eli Shechtman、Fredo Durand 和 Bill Freeman。《改进的分布匹配蒸馏用于快速图像合成》。载于《神经信息处理系统进展》，第 37 卷，第 47455–47487 页，2024a 年。
- Yin 等人（2024b 年）Tianwei Yin、Michaël Gharbi、Richard Zhang、Eli Shechtman、Fredo Durand、William T Freeman 和 Taesung Park。《基于分布匹配蒸馏的单步扩散》。载于《IEEE/CVF 计算机视觉与模式识别会议论文集》，第 6613–6623 页，2024b 年。
- Yin 等人（2025 年）Tianwei Yin、Qiang Zhang、Richard Zhang、William T Freeman、Fredo Durand、Eli Shechtman 和 Xun Huang。《从慢速双向到快速自回归视频扩散模型》。载于《计算机视觉与模式识别会议论文集》，第 22963–22974 页，2025 年。
- You 等人（2026 年）Yuyang You、Yongzhi Li、Jiahui Li、Yadong Mu、Quan Chen 和 Peng Jiang。《自适应视频蒸馏：缓解少步生成中的过饱和与时间坍缩》。arXiv 预印本，编号 2603.21864，2026 年。
- Zewei 和 Yunpeng（2025 年）Tao Zewei 和 Huang Yunpeng。《MagiAttention：面向超长上下文与异构掩码训练的线性可扩展分布式注意力机制》。https://github.com/SandAI-org/MagiAttention/，2025 年。
- Zhang 等人（2025a 年）Jintao Zhang、Haoxu Wang、Kai Jiang、Shuo Yang、Kaiwen Zheng、Haocheng Xi、Ziteng Wang、Hongzhou Zhu、Min Zhao、Ion Stoica 等。《SLA：通过可微调稀疏线性注意力超越扩散 Transformer 中的稀疏性》。arXiv 预印本，编号 2509.24006，2025a 年。
- Zhang 等人 (2025b) Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, 和 Jun Zhu。TurboDiffusion：将视频扩散模型加速 100-200 倍。arXiv 预印本 arXiv:2512.16093，2025b。
- Zhang 等人 (2026) Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang, Ion Stoica, Jianfei Chen, Jun Zhu, 和 Joseph E Gonzalez。SLA2：具有可学习路由和 QAT 的稀疏线性注意力。arXiv 预印本 arXiv:2602.12675，2026。
- Zhao 等人 (2026) Min Zhao, Hongzhou Zhu, Kaiwen Zheng, Zihan Zhou, Bokai Yan, Xinyuan Li, Xiao Yang, Chongxuan Li, 和 Jun Zhu。Causal Forcing++：用于实时交互式视频生成的可扩展少步自回归扩散蒸馏。arXiv 预印本 arXiv:2605.15141，2026。
- Zhao 等人 (2023) Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, 等。PyTorch FSDP：扩展全分片数据并行的经验。arXiv 预印本 arXiv:2304.11277，2023。
- Zheng 等人 (2023a) Kaiwen Zheng, Cheng Lu, Jianfei Chen, 和 Jun Zhu。DPM-Solver-v3：具有经验模型统计数据的改进扩散 ODE 求解器。Advances in Neural Information Processing Systems，36:55502–55542，2023a。
- Zheng 等人 (2023b) Kaiwen Zheng, Cheng Lu, Jianfei Chen, 和 Jun Zhu。扩散 ODE 最大似然估计的改进技术。收录于 International Conference on Machine Learning，第 42363–42389 页。PMLR，2023b。
- Zheng 等人 (2025a) Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, 和 Ming-Yu Liu。DiffusionNFT：基于前向过程的在线扩散强化。arXiv 预印本 arXiv:2509.16117，2025a。
- Zheng 等人 (2025b) Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming-Yu Liu, Jun Zhu, 和 Qinsheng Zhang。直接判别优化：你的基于似然的视觉生成模型秘密地是一个 GAN 判别器。arXiv 预印本 arXiv:2503.01103，2025b。
- Zheng 等人 (2025c) Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, 和 Qinsheng Zhang。掩码扩散模型实际上是时间无关的掩码模型，并利用了不准确的类别采样。发表于国际学习表征会议，第 2025 卷，页码 63186–63227，2025c。
- Zheng 等人 (2025d) Kaiwen Zheng, Yuji Wang, Qianli Ma, Huayu Chen, Jintao Zhang, Yogesh Balaji, Jianfei Chen, Ming-Yu Liu, Jun Zhu, 和 Qinsheng Zhang。通过得分正则化连续时间一致性实现大规模扩散知识蒸馏。arXiv 预印本 arXiv:2510.08431，2025d。
- Zhou 等人 (2024) Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, 和 Hai Huang。得分恒等蒸馏：预训练扩散模型用于一步生成的指数级快速蒸馏。发表于第四十一届国际机器学习大会，2024。
- Zhu 等人 (2026) Hongzhou Zhu, Min Zhao, Guande He, Hang Su, Chongxuan Li, 和 Jun Zhu。因果强制：正确实现自回归扩散蒸馏以实现高质量实时交互式视频生成。arXiv 预印本 arXiv:2602.02214，2026。
- Zou 等人 (2026) Kai Zou, Dian Zheng, Hongbo Liu, Tiankai Hang, Bin Liu, 和 Nenghai Yu。Hiar：通过分层去噪实现高效的自回归长视频生成。arXiv 预印本 arXiv:2603.08703，2026。
附录 A TrigFlow-sCM 与 RF-sCM 的理论分析
本节比较了针对 RF 原生速度预测器的两种连续时间一致性蒸馏实现方式：(i) 对 RF 速度预测器应用 TrigFlow 包装器，然后使用 TrigFlow-sCM 目标函数 (Zheng 等人, 2025d)，以及 (ii) 在 RF 调度下直接编写 sCM 目标函数。尽管不同的扩散噪声调度（例如 TrigFlow 和 RF）在缩放因子范围内是等价且可相互转换的 (Zheng 等人, 2023b)，但我们表明，它们会导致 sCM 的归一化均方误差训练目标函数通常不同。这种差异源于 TrigFlow 包装器的输入输出缩放、切线归一化以及 JVP 的有限精度计算顺序。
RF 和 TrigFlow 坐标。
设 表示 RF 时间，设 表示 TrigFlow 时间。定义
RF 和 TrigFlow 前向过程是
它们通过一个随时间变化的状态缩放相关联：
| (21) |
设 为一个 RF 速度预测器，并设 表示 RF 教师速度。直接 RF 一致性映射为
| (22) |
作为输入-输出变换的 TrigFlow 封装器。
围绕 RF 速度预测器的 TrigFlow 封装器可以写为
| (23) |
等价地，该封装器首先应用输入变换 ，评估 RF 速度预测器，然后应用输出变换
在 TrigFlow 预处理条件下
| (24) |
代入公式 23 得到
| (25) |
因此，在公式 21 的变量变换之后，TrigFlow 封装器和直接 RF 参数化定义了相同的一致性映射。
直接 RF-sCM 切线。
RF 教师 ODE 为
对于停止梯度网络 ，定义 RF JVP
| (26) |
RF 一致性映射的切线为
| (27) |
因此直接 RF-sCM 目标可以写为
| (28) |
在前向值中为零，但它仍然指示相对于哪个输出坐标计算梯度。
通过输入和输出变换的 TrigFlow JVP。
接下来我们在 RF 坐标中重写 TrigFlow-sCM JVP。沿着 TrigFlow 教师轨迹，
| (29) |
使用 和 ，其中
我们得到
| (30) |
由于
显式状态项相互抵消，并且
| (31) |
因此，进入 TrigFlow 封装器内部 RF 速度预测器的 JVP 方向为
| (32) |
在精确算术中，
| (33) |
公式 23 中的输出变换包含显式的 依赖系数。因此，封装后速度的 JVP 为
| (34) |
现在对公式 24 中的 TrigFlow 一致性映射求导：
| (35) |
代入 ，
以及公式 34，我们得到
| (36) |
使用
我们得到紧凑的关系式
| (37) |
这一推导明确表明，TrigFlow 封装器没有引入新的 RF-JVP 结构：在输入和输出变换之后，相同的 RF 组合
出现。在切线层面上，唯一的精确算术差异是因子 。
RF 速度坐标中的 TrigFlow-sCM 目标。
TrigFlow-sCM 目标应用于 TrigFlow 速度坐标中。由公式 23，
| (38) |
因此，对于
其中 如 sCM (Lu and Song, 2024) 和 rCM (Zheng et al., 2025d) 中所取，TrigFlow-sCM 损失变为
| (39) |
相比之下，直接 RF-sCM 是公式 28。因此，这两个目标共享相同的零一致性条件，
但它们通常并不等同于相同的归一化均方误差目标函数。
切向量归一化的影响。
当满足特定条件时，这种区别最容易看出。方程 28 给出了 RF 归一化切向量目标：
方程 39 给出了用 RF 输出坐标表示的 TrigFlow 目标：
因此，如果保持一般性，
关于 RF 速度预测器的损失梯度满足：
| (40) |
当满足特定条件时，这种差异无法简化为简单的标量重新加权，因为 TrigFlow 的分母变为 ，而 RF 的分母是 。因此，切向量归一化打破了两类归一化均方误差损失之间的严格等价性。
有限精度下的 JVP 计算。
方程 33 中的关系仅在实数运算中精确成立。在浮点计算中，
| (41) |
一般情况下，JVP 重排（Lu and Song, 2024）进一步将系数吸收到 JVP 计算中。将缩放因子置于 JVP 方向内部，会使缩放后的切向量传播到网络的每一层；而先将其分解出来，则是先计算未缩放的切向量，然后再对结果进行缩放。由于舍入误差、混合精度转换、融合内核、激活检查点以及自定义 FlashAttention JVP 实现的存在，这两种计算顺序可能产生不同结果。
此外，TrigFlow 封装器包含显式的输入-输出变换项，这些项在代数上精确抵消，但在位级别上不一定精确抵消。例如，方程 37 的推导抵消了来自对 、 以及封装器系数求导的状态相关项。而直接的 RF 实现则计算紧凑表达式：
无需这些中间变换项。因此，即使在精确算术条件下切向量关系成立，在实际的大规模混合精度训练中，这两种实现也不应预期在位级别上等价。这种数值差异会被归一化目标 放大，尤其是当 较小或稳定常数 较小时。
附录 B：带自定义掩码的 FlashAttention-2 JVP 内核
对于 TF-sCM，学生网络在一个打包序列上进行评估，该序列在 TF 注意力掩码下将干净的上下文 token 和带噪声的目标 token 拼接在一起。雅可比向量积（JVP）必须通过与原始前向传播完全相同的掩码注意力算子来计算。密集的加法掩码在概念上很简单，但对于长视频序列来说内存效率低下。因此，我们按照 MagiAttention（Zewei 和 Yunpeng，2025）的风格，将自定义掩码表示为可查询-键矩形的稀疏列表，并仅在 FlashAttention-2 循环内流式传输这些矩形。
设为一个自定义注意力掩码，其中表示查询 token 可以关注键 token，否则为。掩码注意力输出为
| (42) |
对于 JVP，给定切线，得分切线为
| (43) |
掩码是一个离散的路由对象，没有切线。因此，被掩码屏蔽的条目被分配零切线贡献：
等价地，JVP 是通过掩码注意力图计算的
其中是固定的。
对于一行，设表示在允许的键上的掩码 softmax 概率。注意力切线为
| (44) |
其中所有求和都是在下的有效键上进行的。内核在与原始 FlashAttention 计算相同的在线 softmax 过程中计算此表达式。对于一个流式传输的块，定义未归一化的概率
除了标准的 FlashAttention 累加器之外，我们还维护三个 JVP 累加器：
归一化后，切线输出为
| (45) |
其中最后一项中的是归一化后的原始输出。用于原始累加器的相同在线缩放因子也应用于，因此 JVP 在数值上与 FlashAttention-2 softmax 归一化保持一致。
稀疏自定义掩码表示。
自定义掩码被表示为一组查询组及其允许的键范围。每个查询组包含一个连续的查询区间和一个有效的键区间列表。
内核启动任务，其中每个任务内部包含一个查询块。每个任务仅流式处理指定范围内的键。这种范围列表视图同时覆盖了密集/全注意力机制和结构化因果掩码：密集注意力机制包含一个查询组和一个完整的键范围，而教师强制或块因果掩码则被分解为少量完整的查询-键矩形。重要的是，相同的稀疏调度同时用于原始分数及其切线，确保切线对应于前向传播中使用的精确掩码注意力算子。
我们在算法1中给出了完整算法。
Kaiwen Zheng
Guande He
Min Zhao
Jintao Zhang
Huayu Chen
Jianfei Chen
, Chen-Hsuan Lin
, Ming-Yu Liu
, Jun Zhu
, Qianli Ma
UT Austin
zkwthu@gmail.com;
dcszj@tsinghua.edu.cn;
mingyul@nvidia.com
\abscontent
Autoregressive video diffusion with causal diffusion transformers has emerged as a major paradigm for real-time streaming video generation and action-conditioned interactive world models. In this work, we extend rCM, an advanced diffusion distillation framework, to autoregressive video diffusion. The core philosophy of rCM lies in the complementarity between forward and reverse divergences, represented by consistency models (CMs) and distribution matching distillation (DMD), respectively, in diffusion distillation. This philosophy naturally carries over to the autoregressive setting, where teacher-forcing (TF) provides an offline, forward-divergence causal training paradigm, while self-forcing (SF) corresponds to an on-policy, reverse-divergence refinement.
Our contributions are: (1) through extensive experiments, we show that teacher-forcing CM is currently the best complement to self-forcing DMD as an initialization strategy (2) we present the first implementation of teacher-forcing-based continuous-time CMs (e.g., sCM/MeanFlow) for autoregressive video diffusion, enabled by our custom-mask FlashAttention-2 JVP kernel, achieving 10 faster convergence compared to discrete-time CMs (dCMs) (3) we introduce Causal-rCM, a leading, unified, and scalable algorithm-infrastructure open recipe for diffusion distillation and causal training (4) we achieve state-of-the-art streaming video generation performance in both frame-wise and chunk-wise settings, using only synthetic data for training.
Notably, our distilled 2-step causal Wan2.1-1.3B model achieves a VBench-T2V score of 84.63 with only 1 or 2 sampling steps. We further apply Causal-rCM to Cosmos 3, an advanced omnimodal world foundation model for physical AI with action-conditioned generation capability, enabling an interactive world model.
1 Introduction
Video diffusion models are widely recognized as a form of world simulators (Brooks et al., 2024; Bao et al., 2024; Kong et al., 2024; Wan et al., 2025; Ali et al., 2025; Gao et al., 2025; Seedance et al., 2026; NVIDIA, 2026). Instead of denoising all frames jointly with a bidirectional-attention diffusion transformer, autoregressive (AR) video diffusion (Jin et al., 2025; Teng et al., 2025; Chen et al., 2025a) performs next-frame or next-chunk prediction with causal-attention diffusion transformers. This mirrors the shift from masked diffusion (Sahoo et al., 2024; Shi et al., 2024; Zheng et al., 2025c) to block diffusion (Arriola et al., 2025) in the discrete diffusion regime. In this paradigm, the model is autoregressive across frames or chunks, while diffusion denoising is performed within each frame or chunk. This enables streaming long video generation (Huang et al., 2025b; Yang et al., 2026; Chen et al., 2026), interactive world models (Hong et al., 2025; HunyuanWorld, 2025; He et al., 2025; Robbyant Team et al., 2026), and embodied AR video diffusion for closed-loop robot control (Feng et al., 2025; Li et al., 2026c; Ye et al., 2026).
Common causal training paradigms, such as teacher-forcing (TF) and diffusion-forcing (DF) (Chen et al., 2024), suffer from error accumulation and quality degradation over time during AR diffusion inference, commonly known as exposure bias (Schmidt, 2019; Ning et al., 2024). The recent self-forcing paradigm (Huang et al., 2025a; Lin et al., 2025b) resolves this issue by using on-policy training to tackle the training-inference gap, coupled with distribution matching distillation (DMD) (Yin et al., 2024b, a) or adversarial GAN losses (Lin et al., 2025a) for diffusion step distillation. Self-forcing approaches have pushed AR video diffusion toward practical low-latency, real-time, and long-horizon generation in streaming and interactive settings.
However, self-forcing with DMD or GAN objectives is sensitive to initialization and suffers from mode collapse, as DMD-style objectives are based on reverse-KL divergence and optimize student-generated rollouts. Existing AR diffusion systems therefore introduce different initialization strategies before self-forcing, such as ODE-pair regression (Yin et al., 2025; Huang et al., 2025a; He et al., 2025; Zhu et al., 2026), diffusion-forcing-style causal adaptation (Huang et al., 2025b; Robbyant Team et al., 2026), or hybrid TF/DF initialization (Hong et al., 2025). These designs suggest that a stable offline causal objective is crucial before on-policy distribution matching, but the connection between initialization, causal training paradigms, and distillation losses remains underexplored.
In this work, we introduce Causal-rCM, extending rCM (score-regularized consistency model) (Zheng et al., 2025d) to AR video diffusion. In rCM, the key insight is the forward-reverse complementarity at the level of distillation objectives: CMs act as forward-divergence, trajectory-preserving objectives, while DMD acts as a reverse-divergence, distribution-matching objective. In AR diffusion, an analogous complementarity arises at the level of causal training paradigms, where teacher-forcing provides an offline, mode-covering training signal and self-forcing provides an on-policy refinement signal under autoregressive rollouts. Based on this correspondence, Causal-rCM uses teacher-forcing CM for few-step causal distillation on offline causal contexts and teacher trajectories, and self-forcing DMD to directly optimize the inference-time few-step distribution.
Relation to Prior Art
CMs are widely used as initialization or regularization for DMD- and GAN-based diffusion distillation (Lin et al., 2025a; Zheng et al., 2025d). Notably, for AR diffusion, APT2 (Lin et al., 2025b) has adopted teacher-forcing-based CM as initialization for the self-forcing stage, with later theoretical support through the lens of frame-level injectivity (Zhu et al., 2026). Causal-rCM differs from previous works by (1) providing a unified divergence perspective on different causal training paradigms, distillation losses, and their synergy, echoing the high-level principle of rCM; (2) conducting a holistic and systematic investigation of different initialization strategies for self-forcing DMD, uncovering their pros and cons; (3) providing the first implementation of teacher-forcing based continuous-time consistency models (sCM (Lu and Song, 2024), MeanFlow (Geng et al., 2025)) with our custom-mask FlashAttention-2 JVP kernel, achieving 10 faster convergence compared to discrete-time CMs (dCMs); (4) introducing a leading, unified, and scalable algorithm-infrastructure open recipe for diffusion distillation and causal training, achieving state-of-the-art performance in AR diffusion distillation.
Forward-Reverse Objective Complementarity
| Method | Domain | Forward Component (Pretrain / Offline) | Reverse Component (Posttrain / On-policy) | Effect / Takeaway |
| DDO (Zheng et al., 2025b) | diffusion / AR mid-training | diffusion loss on real data | anti-likelihood diffusion loss on self-generated negatives | new record FIDs on ImageNet without auxiliary data/model |
| DiffusionNFT (Zheng et al., 2025a) | diffusion RL | forward-process diffusion objective | reward-ranked positive / negative generated samples | 25 efficiency |
| DDRL (Ye et al., 2025) | diffusion RL | forward-KL / diffusion-loss regularization to offline data | GRPO-style reward optimization on generated rollouts | alleviating reward hacking and diversity collapse |
| rCM (Zheng et al., 2025d) | diffusion distillation | (s)CM loss on data / teacher trajectories | DMD loss on student-generated samples | alleviating mode collapse |
| Causal-rCM | AR diffusion distillation | teacher-forcing CM on offline causal contexts | self-forcing DMD on autoregressive student rollouts | TF-CM initializes SF with causal structure and mode coverage |
Notes. The complementarity can be realized either in a single joint stage or in a forward-to-reverse order across separate stages. We use “on-policy” to emphasize self-generated samples or rollouts; in diffusion RL, such data can be online but off-policy in the strict RL sense.
The broader philosophy of jointly leveraging forward and reverse objectives has appeared across diffusion mid-training, diffusion reinforcement learning, and diffusion distillation. Forward or offline objectives, such as diffusion losses, teacher-forcing losses, and CM losses on real data or teacher trajectories, provide stable training signals and preserve mode coverage. Reverse or on-policy objectives, such as DMD, adversarial losses, and reward-driven optimization on generated samples, directly improve the generated distribution but are more sensitive to initialization and coverage. As summarized in Table 1, recent methods including DDO (Zheng et al., 2025b), DiffusionNFT (Zheng et al., 2025a), DDRL (Ye et al., 2025), and rCM (Zheng et al., 2025d) all benefit from this complementarity. Causal-rCM instantiates the same principle in AR diffusion distillation: teacher-forcing CM serves as the forward/offline component, while self-forcing DMD serves as the reverse/on-policy component.
2 Background
2.1 Diffusion Models
Diffusion models (DMs) (Ho et al., 2020; Song et al., 2020) learn continuous data distributions by gradually perturbing clean data with Gaussian noise, which generates a trajectory along with associated marginals , and then learning to reverse this process. The forward process follows a closed-form transition kernel with predefined noise schedule , enabling reparameterization as . The sampling process of DMs can follow the probability flow ordinary differential equation (PF-ODE) , where , , and is the score function (Song et al., 2020). A key property of DMs is the theoretical equivalence of different parameterizations: the network may predict the score (), the noise (), the clean data (), or the velocity (), with optimal predictors being analytically interconvertible (Zheng et al., 2023b). With velocity parameterization (Zheng et al., 2023b), DMs are trained by minimizing the mean square error (MSE) , where the regression target is (denote ), and the PF-ODE is simplified to , commonly known as flow matching (Lipman et al., 2022). A notable special case, rectified flow (RF) (Liu et al., 2022), employs the schedule , which simplifies the velocity target to .
2.2 Diffusion Distillation
Consistency Distillation
Consistency models (CMs) (Song et al., 2023) aim to learn a consistency function which maps the point at arbitrary time on the teacher PF-ODE trajectory to the initial point . Given a free-form student network , the consistency function is usually parameterized as with and (e.g., under the RF schedule). This parameterization naturally satisfies the boundary condition . Here, is the direct counterpart of the data predictor (denoiser) in DMs, while corresponds to the velocity predictor .
The CM objective enforces consistent student outputs at adjacent timesteps and along the teacher trajectory. Discrete-time CMs (dCMs) minimize the following objective with :
| (1) |
where is a positive weighting function, is a distance metric, is the stop-gradient version of , and is obtained by solving the teacher PF-ODE from to with numerical solvers.
Continuous-time CMs (sCM) (Lu and Song, 2024) take the limit in dCM to obtain a more accurate objective. When , the instantaneous CM loss becomes , where is the tangent of at along the teacher ODE trajectory . This tangent can be efficiently computed by forward-mode automatic differentiation, Jacobian-vector product (JVP): . sCM further applies MSE reformulation and tangent normalization, reducing the loss to
| (2) |
MeanFlow (Geng et al., 2025) can be viewed as combining sCM with consistency trajectory models (CTMs) (Kim et al., 2023) under the RF schedule. CTMs extend CMs by adding another time condition and defining a consistency trajectory function , which maps the point to a less noisy point on the teacher ODE trajectory. The infinitesimal jump from to , i.e., , reduces to the diffusion denoiser and serves as an anchor for applying the diffusion loss. This anchor enhances training stability, preserves multi-step sampling, and enables training few-step models from scratch. Thus, CTMs can be viewed as an interpolation between DMs and CMs. In the continuous-time case, CTMs can be optimized with an objective similar to sCM:
| (3) | ||||
Under the RF schedule, we have , and . Since is the velocity predictor , if we take as the ground-truth velocity , then
| (4) |
which recovers the MeanFlow objective. Alternatively, we can set and use the same formulation for distillation, rather than training from scratch.
Distribution Matching Distillation
Distribution matching distillation (DMD) (Yin et al., 2024b, a) is a simple and effective type of score distillation (Wang et al., 2023; Zhou et al., 2024). Given a few-step student generator , with prior distribution , DMD aims to match the student distribution with the teacher distribution by minimizing the reverse-KL divergence on their diffused marginals:
| (5) |
The gradient of this objective can be written as a score difference between the student and teacher distributions:
| (6) |
The teacher score is provided by the pretrained DM, while the student score is intractable for a few-step generator. DMD therefore trains an auxiliary fake score network on student-generated samples with , which serves as a proxy for the student score. In denoiser parameterization, the score difference can be written, up to a time-dependent scalar absorbed into , as the difference between the fake and teacher denoisers. With the adaptive normalization trick in DMD, the student can be updated with the following stop-gradient MSE objective:
| (7) |
DMD alternates between student () and critic () phases, forming an adversarial training dynamic similar to GANs.
2.3 Autoregressive Video Diffusion
Autoregressive (AR) video diffusion factorizes video generation along the temporal dimension. Given a video latent sequence divided into frames or chunks, an AR model generates each block conditioned on previous blocks: Within each temporal block, the model still performs diffusion denoising, e.g., under the RF schedule, with velocity target . Different from bidirectional video diffusion, which denoises all frames jointly with full temporal attention, AR video diffusion uses causal attention so that each frame or chunk only attends to past context. This enables KV caching like LLMs and makes the model naturally suitable for streaming and interactive generation.
Fig. 3 illustrates the three causal training paradigms: teacher-forcing (TF), diffusion-forcing (DF), and self-forcing (SF).
In TF, the model predicts the current noisy block while attending to clean ground-truth history, i.e., . TF is stable and parallelizable via a specific attention mask, but it creates a training-inference gap: during inference, the model must condition on its own generated history rather than ground-truth context.
DF assigns independent noise levels to different frames or chunks and trains the model under a block-causal attention mask, i.e., . This exposes the model to noisy histories and improves robustness. However, the training-inference gap remains: perturbing ground-truth videos with synthetic noise does not match the errors and artifacts accumulated from model-generated rollouts at inference.
SF directly simulates AR inference during training. The student rolls out chunks sequentially with KV caching, , and the loss is applied to the self-generated video. Therefore, SF trains the model under its own inference-time context distribution, directly addressing the exposure bias induced by the training-inference gaps in TF and DF. SF must be combined with reverse-type on-policy objectives, such as DMD or GAN losses.
3 Causal-rCM: A Leading, Unified and Scalable Algorithm-Infrastructure Open Recipe for Diffusion Distillation and Causal Training
3.1 Algorithms
To extend rCM to autoregressive diffusion, we pair its two distillation objectives (CM, DMD) with two causal training paradigms, teacher-forcing (TF) and self-forcing (SF), respectively. This preserves the forward-reverse correspondence of rCM in the autoregressive setting: TF-CM provides an offline, forward-type consistency objective, whereas SF-DMD provides an on-policy, reverse-type distribution-matching objective.
TF-CM requires an autoregressive diffusion teacher that is evaluated under the same clean-context setting as the student during TF-based distillation. Such a causal teacher can be trained from scratch, or adapted from a pretrained bidirectional diffusion model, with TF or DF. It is arguably more reasonable to use TF because it exposes the teacher to clean historical frames, matching the context distribution used in TF-based distillation. The CM component can be instantiated either as the simple dCM or as more advanced continuous-time variants such as sCM and MeanFlow. For SF-DMD, following prior work (Huang et al., 2025a; Lin et al., 2025b), we use a bidirectional teacher and a bidirectional fake-score network to provide real and fake score estimates on self-generated rollouts (Fig. 3(c)) and apply the DMD loss (Eqn. 7).
Unlike rCM, which combines CM and DMD in a joint-training style, Causal-rCM applies TF-CM and SF-DMD sequentially. The full pipeline consists of three stages: (1) TF converts the bidirectional diffusion model into an autoregressive diffusion model, which serves as both the causal teacher and the student initialization for the subsequent TF-CM stage; (2) TF-CM distills the causal teacher into a few-step causal student, which serves as the student initialization for the subsequent SF-DMD stage; and (3) SF-DMD refinement further optimizes the student on its own autoregressive rollouts, reducing the training-inference gap and exposure bias. As summarized in Fig. 4, Causal-rCM provides a simple and strong recipe that avoids cumbersome ODE-pair knowledge distillation (KD) (Luhman and Luhman, 2021) and GAN-style post-training, while introducing a novel TF-sCM implementation and achieving state-of-the-art performance.
3.1.1 Teacher-Forcing, Teacher-Forcing dCM and Self-Forcing DMD
The core operation of TF-based training is to replace a standard single-state forward with a packed causal forward over concatenated clean context and noisy targets. Concretely, for a velocity predictor, instead of evaluating , we evaluate
| (8) |
where is the TF attention mask, the clean part provides ground-truth causal context at timestep , and the loss is applied only to the noisy part. The mask ensures that each noisy block attends only to its allowed clean history and its own noisy tokens, matching the TF pattern in Fig. 3. Such TF-mask attention can be implemented with custom-mask attention operators such as FlexAttention (Dong et al., 2024) or MagiAttention (Zewei and Yunpeng, 2025). An alternative is a two-pass implementation: first cache the clean tokens under a block-causal attention mask, and then perform a second forward pass in which noisy tokens attend to the cached clean context. However, this design requires the clean-token KV cache to be retained in the computational graph, making it less compatible with activation checkpointing and more memory-intensive.
With a diffusion regression target under the RF schedule, the ordinary TF objective is
| (9) |
This gives a full-step causal diffusion model. For TF-dCM, the clean context remains fixed, while the noisy part is moved along the causal teacher PF-ODE trajectory. Let be obtained by solving the causal teacher ODE from at to under the same TF mask. The student minimizes
| (10) | ||||
SF-DMD is applied after TF-CM. The student first performs a temporal AR rollout with KV caching. At chunk , the model generates the current clean chunk conditioned on the cached states of previous generated chunks:
| (11) |
After is generated, it is fed once more into the causal transformer through a cache-update forward pass, which appends its clean-token key/value states to the cache:
| (12) |
Within each chunk, is implemented by few-step self-rollout denoising from pure noise :
| (13) |
In each training iteration, the number of simulation steps is randomly sampled from . Each transition can be instantiated as CM-style reverse denoising followed by forward noising, e.g., under the RF schedule,
| (14) |
The final output enters the DMD loss. Following standard practice (Yin et al., 2024a; Huang et al., 2025a), we apply gradient truncation to make SF-DMD memory-efficient. The intermediate denoising steps and previous-chunk KV caches are detached (indicated by ). Only the final denoising step of each chunk is kept differentiable (indicated by ), which the DMD loss is back-propagated through.
3.1.2 JVP-based Causal Distillation with Teacher-Forcing sCM/MeanFlow
TF-sCM uses the same packed causal forward as TF and TF-dCM, but replaces the finite-step consistency target with a continuous-time tangent target. The clean context is kept fixed, while the noisy tokens move along the causal teacher ODE. Under the RF schedule, define the causal teacher velocity on the noisy branch as
| (15) |
The RF consistency map on the noisy branch is
| (16) |
Its continuous-time tangent along the causal teacher trajectory is
| (17) | ||||
Here the JVP is computed through the same TF-masked packed forward as the primal prediction. The tangent of the clean context is zero, and only the noisy branch follows the teacher velocity.
The TF-sCM objective is then
| (18) |
| (19) |
A subtle but important design choice is to use the RF-native form of sCM, rather than wrapping the RF velocity model into TrigFlow and applying the TrigFlow-sCM objective as in rCM (Zheng et al., 2025d). Although different diffusion noise schedules, such as TrigFlow and RF, are analytically convertible up to a time-dependent scaling (Zheng et al., 2023b), they generally induce different normalized MSE objectives for sCM (Appendix A). In the bidirectional setting, rCM finds the TrigFlow wrapper beneficial for stability. However, in our causal TF setting, the TrigFlow-wrapped TF-sCM results in degraded generation quality, whereas the RF-native TF-sCM produces more smooth outputs.
3.1.3 Extension to Noisy Context and Custom Step Schedule
Noisy context and custom step schedules (Liu et al., 2026) are two simplest and most effective inference acceleration techniques for AR video diffusion distillation. Both TF and SF can naturally incorporate them, as illustrated in Fig. 5.
Noisy Context
Unlike LLMs, AR video diffusion must maintain a denoising-time-aware KV cache: standard clean-context AR inference requires an additional clean-context encoding pass after the denoising steps of each chunk, so an -step causal diffusion model effectively costs number of function evaluations (NFEs) per chunk. Noisy context removes this extra pass by reusing the KV states from the last denoising step as the context for subsequent chunks, reducing the effective latency from to NFEs. Besides acceleration, noisy context can improve long-horizon robustness, as residual noise acts as a low-pass filter that suppresses accumulated high-frequency artifacts while preserving coarse motion dynamics (Huang et al., 2025b).
In TF, noisy context is incorporated by replacing the clean history in the packed TF forward with noisy historical tokens at the corresponding context timestep, while the loss remains applied only to the current target block. In SF, noisy context is used directly during AR rollout. Although introducing noisy context in the TF stages would better align with inference, we find it sufficient in practice to apply it only in the final SF stage.
Custom Step Schedule
The number of denoising steps can also vary across chunks. In text-to-video generation, the first chunk is typically more demanding because it establishes the global scene, layout, and appearance, whereas later chunks mainly extend the video conditioned on previous context. We therefore allow a chunk-dependent step schedule
| (20) |
where denotes the number of denoising steps for chunk . For example, a nominal 2-step model can use , allocating extra computation only to the first chunk.
For SF-DMD training, we cycle the rollout length by the training iteration. For example, for a target schedule , SF-DMD repeatedly cycles through . This cycling strategy is important because SF-DMD only back-propagates through the final denoising step of each chunk. Cycling the rollout length makes different denoising intervals appear as the final differentiable step across iterations, rather than supervising only the last interval of the maximum-step sampler.
3.2 Infrastructure
| Codebase | Recipe Scope | Algorithmic Recipes | Bidirectional Infra | Causal Infra | |||||||||||
| Bi. | Causal | TF | DF | SF | Replayed | FSDP2 | CP/SP | SAC | JVP | FSDP2 | CP/SP | SAC | JVP | KV Cache | |
| Self-Forcing (Huang et al., 2025a) | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | v1 | ✗ | AC | ✗ | ✓post |
| FastVideo (Hao-AI Lab, 2026) | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓F-U | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓post |
| FastGen (Nie et al., 2026a) | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | AC | ✗ | ✓ | ✗ | AC | ✗ | ✓post |
| (Causal-)rCM | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓F-U | ✓ | ✓ | ✓ | ✓F-U | ✓ | ✓ | ✓pre/post |
Notes. ✓: supported; ✗: not found; : partial, unclear, or path-dependent.
TF: teacher-forcing implemented as [clean frames, noisy frames] concatenation with a special causal mask.
DF: diffusion-forcing with ordinary block-causal masking.
SF: self-forcing with self-rollout / KV-cache-style training execution.
Replayed: replayed back-propagation technique that avoids storing the entire computation graph during self-rollout.
FSDP2: Fully Sharded Data Parallel v2. v1: FSDP1-only support.
CP/SP: context/sequence parallel. T: temporal/frame axis; F: flattened video-token axis, e.g., flattened THW patch tokens. U: DeepSpeed-Ulysses; R: Ring Attention; UR: Ulysses–Ring hybrid (USP).
SAC: selective activation checkpointing. AC: activation checkpointing, but not clearly op-level SAC.
JVP: Jacobian-vector-product. Base operator for continuous-time consistency model (sCM/MeanFlow).
KV Cache: causal self-attention KV cache. pre: K is cached before RoPE; post: K is cached after RoPE.
Causal-rCM is designed as an algorithm-infrastructure recipe. Its main infrastructure goal is to make causal training paradigms (TF, DF, and SF), continuous-time JVP-based CMs, and large-scale parallel training mutually compatible. Achieving this requires careful co-design of attention-mask specification, KV caching, FSDP2, context parallelism, activation checkpointing, FlashAttention-2 JVP kernels, and replayed back-propagation. Table 2 summarizes the resulting system-level coverage and highlights the infrastructure advantages of Causal-rCM over other widely used codebases.
3.2.1 Main Components
FlashAttention-2 JVP Kernel with Custom Masks
Continuous-time CMs require the tangent of the network output along the teacher ODE. Computing this tangent with a generic torch.func.jvp over unfused attention is impractical for large video transformers due to the materialization of large attention intermediates and the resulting memory overhead. To enable JVP through fused attention under TF masks, we build on the FlashAttention2-JVP kernel in rCM (Zheng et al., 2025d) and extend it to support custom masks. The TF mask is represented as admissible query-key ranges rather than materialized dense matrices. The details are presented in Appendix B.
Parallelisms
We use FSDP2 (Zhao et al., 2023) as a ZeRO-3-style sharding backend: parameters, gradients, and optimizer states are partitioned across data-parallel ranks, and each module materializes full parameters only for its local computation. This reduces per-GPU model-state memory and makes it feasible to train large video DiTs with student, teacher, fake-score, and EMA networks in the same distillation pipeline. We use distributed checkpointing (DCP) to save and restore the sharded model and optimizer states directly across ranks, avoiding the need to gather full model states on a single process.
We use flattened Ulysses-style context parallelism (CP) (Jacobs et al., 2023) to shard the long video-token sequence across ranks. Specifically, the spatiotemporal video tokens are first flattened into a single sequence, and CP partitions this flattened sequence dimension across P devices. Before attention, each GPU holds a shard of size [B, H, L/P, C] for QKV. An all-to-all operation then redistributes QKV to [B, H/P, L, C] for local attention, followed by another all-to-all to restore the sequence partition of the attention output . A key design choice is to make CP transparent to the outer algorithm: the network interface always takes and returns the global full sequence, independent of CP size, while the network internally handles local sequence shards, all-to-all attention, and output gathering.
Activation Checkpointing
We use selective activation checkpointing (SAC) to reduce activation memory by recomputing only selected parts of the network during backward. Unlike vanilla region-based torch.utils.checkpoint, SAC provides finer-grained control over which operations are recomputed and which intermediates are preserved. In practice, we apply SAC mainly to compute-heavy stateless regions such as attention and MLP blocks, while leaving lightweight or stateful operations outside checkpointed regions.
KV Cache
The KV cache is used by causal rollout execution and inference. We distinguish three cache modes: disabled mode for ordinary packed training, append mode for committing a generated chunk into the cache, and readonly mode for generating the current chunk while attending to previously committed chunks. Cached K/V tensors are detached by construction, which prevents gradients from propagating through previous chunks and keeps SF-DMD memory bounded. The cache also records chunk boundaries, so a readonly forward can expose only the prefix needed by the current block. This supports both standard AR rollout and variants such as noisy context, where the final denoising forward can be reused as the context state.
We support both pre-RoPE and post-RoPE key caching. Post-RoPE caching is simple and efficient because cached keys can be reused directly. Pre-RoPE caching is useful when the same cached content may need different positional treatment, e.g., for length extrapolation or alternative position indexing (Yesiltepe et al., 2026; Yi et al., 2025; Li et al., 2026b; Kim et al., 2026). The implementation keeps this choice inside the attention context so that the high-level rollout code does not need to distinguish the two cases.
Replayed Back-propagation
SF-DMD generates on-policy videos through AR rollout. In the standard execution with gradient-truncation, the final differentiable denoising steps of all chunk are kept in the computational graph, which can be memory-intensive for long videos. We therefore provide an optional replayed back-propagation mode (Hong et al., 2025) as a memory-saving implementation. The rollout is first constructed without gradients, while storing the final noisy input, timestep, detached KV cache, and DMD target for each chunk. Then, each chunk’s final denoising step is recomputed with gradients enabled, and its gradient is back-propagated separately with gradient accumulation. This trades additional computation for lower activation memory. We deliberately reserve this replayed path for SF-DMD: TF, DF, and TF-CM remain packed, since replaying differentiable prefix-KV computation offers limited additional benefit once SAC is enabled.
3.2.2 Compatibility Design
A major goal of Causal-rCM is to make advanced causal training features composable. In practice, many components that work independently can conflict when used together. We therefore implement compatibility at the level of execution semantics rather than as independent feature switches.
SAC FlexAttention.
Packed TF/DF/TF-CM training relies on custom-mask attention. In the FlexAttention path, the attention pattern is specified by a mask_mod function and lowered by the PyTorch compiler into a specialized fused attention kernel. To make this compatible with SAC, we use torch>=2.10 together with
| torch._inductor.config.wrap_inductor_compiled_regions = True |
which exposes Inductor-compiled FlexAttention calls to SAC as explicit checkpointable regions, internally represented as inductor_compiled_code.
SAC self-forcing.
SF-DMD rollout is stateful because KV caches and causal metadata evolve across chunks. We make this compatible with SAC by separating persistent cache storage from per-forward causal state: historical K/V tensors are stored as detached context, while each forward constructs a fresh CausalInferenceState describing the current chunk, cache range, and append/read-only mode for future recomputation. The inference state is not reused through in-place updates, so checkpoint recomputation reconstructs the same causal context as the original forward. Cache-append forwards are kept outside checkpointed execution, so recomputation never replays cache mutation; checkpointed regions only read a fixed causal context.
JVP FSDP2.
Following rCM, we implement JVP at the layer level, rather than applying a global torch.func.jvp to an FSDP2-wrapped model. Each layer exposes a paired primal-tangent interface, taking as input and returning . This corresponds to an FSDP2(JVP) design instead of JVP(FSDP2). FSDP2 continues to manage parameter materialization, sharding, and gradient reduction at layer boundaries, while tangent propagation is performed locally within each layer’s forward computation.
JVP Ulysses CP.
Ulysses CP extends naturally to JVP because tangent tensors follow the same communication pattern as their primal counterparts. Specifically, are all-to-all exchanged together with , the local attention computation is replaced by our custom-mask FlashAttention-2 JVP kernel, and the resulting is returned through the same output all-to-all as . We reuse the JVP-compatible distributed-attention design from rCM, while adding custom-mask support for packed TF/DF/TF-CM training.
KV cache Ulysses CP.
For rollout execution, cached K/V tensors must be compatible with Ulysses CP. We use a post-all-to-all KV cache, where the cache is stored in the same [B, H/P, L, C] layout as exposed to local attention. Each CP rank directly reuses its head-sharded, full-sequence cached K/V states. This avoids repeatedly converting old cache entries between global and CP-local layouts.
4 Experiments
4.1 Setup
| Configuration | Stage 1 | Stage 2 | Stage 3 | ||
| Wan2.1-1.3B TF/DF | Wan2.1-14B TF/DF | Wan2.1-1.3B TF-dCM | Wan2.1-1.3B TF-sCM | Wan2.1-1.3B SF-DMD | |
| Global batch size | 256 | 64 | 32 | 32 | 64 |
| Context parallel size | 1 | 8 | 4 | 4 | 4 |
| Student optimizer | AdamW lr wd | AdamW lr wd | AdamW lr wd | AdamW lr wd | AdamW lr wd |
| Fake-score optimizer | – | – | – | – | AdamW lr wd |
| CFG scale | – | – | 3.0 | 3.0 | 5.0 |
| Time sampling / weighting | TF: , shared , Gaussian-bell weight; DF: , random per-chunk , no weight | TF: , shared , Gaussian-bell weight; DF: , random per-chunk , no weight | uniform RF grid with shift , steps , skip | ||
| Specific hyperparameters | – | – | – | tangent warmup | max rollout steps student update freq. |
| Training iterations | 30k | 30k | 10k | 1k | varies |
Models and Datasets.
We conduct the main streaming video generation experiments on Wan2.1 T2V (Wan et al., 2025) at 480p resolution. Videos are generated at spatial resolution with 81 RGB frames, corresponding to 21 latent frames after VAE temporal compression. Training uses the synthetic T2V data provided by rCM (Zheng et al., 2025d), generated by the bidirectional Wan2.1-14B teacher with 100-step Euler sampling, shift 3.0, and CFG scale 5.0. We use Wan2.1-1.3B as the main student model and use Wan2.1-14B teachers for distillation.
We evaluate two causal chunk patterns. The frame-wise setting, denoted by c1-1, uses one initial latent frame and then one-latent-frame chunks. The chunk-wise setting, denoted by c3-3, uses one initial latent chunk of three frames and then three-latent-frame chunks. The same chunk pattern is used consistently for packed TF/DF/TF-CM masks, SF-DMD rollout, KV-cache inference, and streaming evaluation.
Training.
Causal-rCM uses a three-stage training recipe. We report the main hyperparameters in Table 3. For TF-CM, we use 14B causal teachers trained with TF. For SF-DMD, we use 14B birectional teacher and fake score networks.
For few-step SF-DMD, we use RF sampling schedules with a maximum of 4 denoising steps. The 4-step sampler uses intermediate times . The 2-step sampler uses 4 steps for the first chunk and 2 steps for later chunks, with schedule . The 2-step noisy-context variant uses schedule and reuses the final denoising forward as the context cache. The 1-step variant uses 4 steps for the first chunk and 1 step for later chunks, with schedule .
Evaluation Metrics.
For streaming quality, we evaluate text-to-video generation with VBench-T2V (Huang et al., 2024), reporting the total score as well as the quality and semantic sub-scores.
For inference efficiency, we report the number of function evaluations (NFE), throughput in frames per second (FPS), first-chunk latency, and second-chunk latency. All efficiency measurements are conducted with batch size 1 on a single H100 GPU. The reported FPS and latency include both diffusion sampling and VAE decoding.
4.2 Results
4.2.1 Streaming Video Generation
Main Results.
Table 4 compares Causal-rCM against bidirectional Wan2.1 and streaming video generation baselines, including Self-Forcing (Huang et al., 2025a), LongLive (Yang et al., 2026), Causal Forcing (Zhu et al., 2026), and AnyFlow (Gu et al., 2026). We report both frame-wise and chunk-wise results. Causal-rCM achieves state-of-the-art streaming quality while supporting 4-step, 2-step, 2-step noisy-context, and 1-step inference schedules.
| Method | NFE | Total Score | Quality Score | Semantic Score | Throughput | First Latency | Second Latency | SF-DMD iters |
| (FPS) | (s) | (s) | ||||||
| Bidirectional | ||||||||
| Wan2.1-1.3B | 502 | 82.78 | 83.44 | 80.13 | 0.72 | – | – | – |
| Wan2.1-14B | 502 | 83.35 | 83.97 | 80.88 | 0.18 | – | – | – |
| Frame-wise (c1-1) | ||||||||
| Causal Forcing (4-step) | 5 | 81.56 | 82.59 | 77.44 | 8.3 | 0.40 | 0.46 | – |
| Causal-rCM (4-step) | 5 | 84.29 | 85.27 | 80.36 | 8.3 | 0.40 | 0.46 | 1200 |
| Causal-rCM (2-step) | 3 | 84.63 | 85.46 | 81.31 | 12.2 | 0.40 | 0.31 | 3000 |
| Causal-rCM (2-step, noisy ctx) | 2 | 83.11 | 83.55 | 81.37 | 15.9 | 0.40 | 0.23 | 1500 |
| Causal-rCM (1-step) | 2 | 84.63 | 85.54 | 81.01 | 15.9 | 0.40 | 0.23 | 3000 |
| Chunk-wise (c3-3) | ||||||||
| Self-Forcing (4-step) | 5 | 83.76 | 84.53 | 80.68 | 17.4 | 0.57 | 0.64 | – |
| LongLive (4-step) | 5 | 83.62 | 84.36 | 80.69 | 17.4 | 0.57 | 0.64 | – |
| Causal Forcing (4-step) | 5 | 83.96 | 84.94 | 80.04 | 17.4 | 0.57 | 0.64 | – |
| AnyFlow (4-step) | 5 | 84.31 | 85.15 | 80.94 | 17.4 | 0.57 | 0.64 | – |
| Causal-rCM (4-step) | 5 | 84.37 | 85.02 | 81.73 | 17.4 | 0.57 | 0.64 | 1250 |
| Causal-rCM (2-step) | 3 | 84.30 | 85.04 | 81.36 | 22.2 | 0.57 | 0.49 | 2500 |
| Causal-rCM (2-step, noisy ctx) | 2 | 84.24 | 84.96 | 81.36 | 25.6 | 0.57 | 0.41 | 1750 |
| Causal-rCM (1-step) | 2 | 84.01 | 84.71 | 81.22 | 25.6 | 0.57 | 0.41 | 3000 |
Performance under Custom Step Schedule and Noisy Context.
Table 4 shows an interesting behavior under custom step schedules. In the frame-wise setting, the 1-step and 2-step Causal-rCM models outperform the 4-step variant, which is counter-intuitive at first glance. We attribute this to the nature of the frame-wise setting: each AR chunk contains only a single latent frame and therefore has no internal temporal structure to denoise. In this case, allocating many denoising steps to every future chunk can over-emphasize autoregressive feedback errors, especially considering the gradient truncating strategy of SF-DMD. Empirically, we observe that 4-step frame-wise SF-DMD is more prone to camera drift, e.g., a consistent leftward camera rotation across samples, and can only be trained stably for about 1k iterations. In contrast, using 1 or 2 steps for later chunks largely suppresses this drift and allows stable training for around 3k iterations. Since each future chunk contains only one latent frame, 1–2 denoising steps are already sufficient to generate the frame, and the reduced rollout depth improves stability.
The trend is different in the chunk-wise setting, where each chunk contains three latent frames and therefore has non-trivial internal temporal correlation. Here, a deeper 4-step sampler provides a better denoising trajectory for modeling motion and intra-chunk consistency, leading to the best overall score. This suggests that the optimal step schedule depends on the temporal span of each AR chunk: frame-wise generation benefits more from shallow, stable rollout, while chunk-wise generation benefits from additional denoising depth.
Noisy context further improves inference efficiency by eliminating the extra clean-context KV encoding pass, reducing the effective cost from to NFEs per chunk. Comparing 2-step sampling with noisy context against 1-step sampling, we find that 1-step sampling is better in the frame-wise setting, while 2-step sampling with noisy context is better in the chunk-wise setting. This is consistent with the above observation. For single-frame chunks, the extra denoising step brings limited benefit, while the residual noise in the context can directly affect fine-grained details in frame-level prediction. For three-frame chunks, the chunk contains a higher-dimensional and more redundant spatiotemporal token group. In this regime, Gaussian perturbations are less likely to destroy the entire chunk-level structure uniformly, and much of the motion and coarse semantic context can still be preserved (Hoogeboom et al., 2023). Therefore, 2-step sampling with noisy context can retain the benefit of an additional denoising step for intra-chunk temporal coherence.
Comparison between TF-dCM and TF-sCM.
Fig. 6 compares TF-dCM and TF-sCM before the final SF-DMD stage. TF-sCM consistently provides a stronger initialization with over fewer training iterations. In the frame-wise setting, TF-sCM reaches above 81.8 VBench-T2V score within 1-2k iterations, already surpassing TF-dCM trained for 10k iterations. The gap is even clearer in the chunk-wise setting, where TF-sCM reaches above 83 within 1-2k iterations, while TF-dCM improves more slowly and remains lower after much longer training.
Ablation Studies on Initialization Strategies.
| Initialization | Total Score | Quality Score | Semantic Score | SF-DMD iterations |
| Frame-wise (c1-1) | ||||
| DF | 83.11 | 83.85 | 80.16 | 2000 |
| TF | 82.62 | 83.62 | 78.61 | 1000 |
| DF-KD | 80.59 | 80.41 | 81.32 | 2000 |
| TF-KD | 83.49 | 84.50 | 79.43 | 1250 |
| TF-dCM | 84.29 | 85.27 | 80.36 | 1200 |
| TF-sCM | 83.84 | 84.67 | 80.55 | 1000 |
| Chunk-wise (c3-3) | ||||
| DF | 84.80 | 85.58 | 81.65 | 1500 |
| TF | 84.95 | 85.82 | 81.47 | 1000 |
| DF-KD | 83.61 | 84.10 | 81.68 | 1500 |
| TF-KD | 83.79 | 84.41 | 81.30 | 1000 |
| TF-dCM | 84.33 | 85.22 | 80.75 | 3200 |
| TF-sCM | 84.37 | 85.02 | 81.73 | 1250 |
Table 5 ablates the initialization strategies of SF-DMD. We compare causal diffusion initializations from DF and TF, ODE-pair knowledge distillation variants (DF-KD and TF-KD), and teacher-forcing consistency initializations (TF-dCM and TF-sCM). The corresponding training curves are shown in Fig. 7.
In the frame-wise setting, TF-CM initialization achieves the best overall performance, with DF and TF-KD also providing competitive alternatives. Although TF-sCM starts from a stronger initial model, TF-dCM is more stable during SF-DMD and supports longer refinement, leading to a higher peak score. In the chunk-wise setting, DF/TF initialization achieves the highest VBench-T2V scores, close to 85. However, as shown in Fig. 8, these models often produce over-smoothed and over-saturated textures, such as water, hair, and leaves, with noticeably fewer fine-grained details. Considering both VBench scores and qualitative inspection, TF-CM initialization is still the most reliable choice. Among the two TF-CM variants, TF-sCM slightly outperforms TF-dCM while requiring fewer SF-DMD iterations.
4.2.2 Interactive World Model
We further apply Causal-rCM to Cosmos 3 (NVIDIA, 2026), an omnimodal world model based on a two-tower Mixture-of-Transformers architecture. Cosmos 3 separates an understanding tower (UND) for text and prompt reasoning from a generation tower (GEN) for vision, action, and sound tokens, while sharing the multimodal attention layers and unified 3D mRoPE across modalities. In the original generator mode, GEN tokens use bidirectional self-attention for multimodal denoising. To support interactive world modeling, we convert the GEN vision stream into a temporal-causal autoregressive diffusion stack (Fig. 9).
We treat each latent video frame as a vision supertoken, which contains all spatial latent tokens of that frame. Temporal-causal attention is applied at the supertoken level: future vision supertokens are masked from past and current ones, while spatial tokens within the same vision supertoken remain fully bidirectional.
The same causal stack supports text-to-video, image-to-video, and forward-dynamics (action-conditioned) modeling. In text-to-video, all vision supertokens are generated from text conditioning. In image-to-video and forward dynamics, the first vision supertoken is provided as clean context, and the model predicts future vision supertokens autoregressively. For forward dynamics, action supertokens are treated as input conditions. A null action supertoken is used for the first frame, and real action supertokens are aligned by unified 3D mRoPE to the next generated vision supertoken, so that action controls the transition from state to .
As shown in Fig. 10, the interactive Cosmos 3 model supports streaming control: given the same initial scene, the generated future frames follow distinct trajectories under left-turn, right-turn, and stay-forward controls.
5 Related Work
Differential information and JVPs in generative modeling.
Differential information has played an important role in diffusion ODEs beyond standard first-order denoising supervision. High-order denoising score matching shows that first-order score matching is insufficient for maximum-likelihood diffusion ODE training, and controls higher-order score errors to tighten the likelihood gap (Lu et al., 2022). Subsequent work improves diffusion ODE likelihood estimation and training with velocity parameterization, variance reduction, and high-order flow-matching objectives (Zheng et al., 2023b). DPM-Solver-v3 further uses empirical model statistics of a pretrained diffusion model to derive improved ODE solver coefficients, and also reveals numerical issues related to time derivatives in diffusion networks (Zheng et al., 2023a). More recently, sCM, MeanFlow, AYF, and FACM use JVPs as a direct training signal for continuous-time consistency or flow-map objectives (Lu and Song, 2024; Geng et al., 2025; Sabour et al., 2025; Peng et al., 2025). rCM scales JVP-based consistency distillation to large image and video diffusion models by making JVP computation compatible with FlashAttention, FSDP, and context parallelism, and combines it with DMD regularization (Zheng et al., 2025d). Causal-rCM extends this line to autoregressive video diffusion, applying JVP-based teacher-forcing sCM under clean causal contexts as a structured initialization for self-forcing DMD.
Forward-reverse complementarity in distillation objectives.
A growing set of few-step methods can be viewed as combining a coverage-preserving forward component with a quality- or reward-seeking reverse component. For text-to-image generation, recent practical studies standardize large-scale few-step distillation recipes for strong text-conditioned teachers, and empirically compare sCM with MeanFlow (Pu et al., 2025). Flow-map methods distill teacher ODE behavior more directly. FreeFlow (Tong et al., 2025) performs data-free flow-map distillation by sampling from the prior and querying teacher dynamics on student-induced flow-map states, with an additional correction objective to mitigate compounding errors. In contrast, -Flow (Chen et al., 2025b) is more explicitly on-policy by matching teacher velocities along the student policy’s own ODE trajectory. Distribution-matching methods improve few-step quality but may sacrifice diversity; recent variants therefore introduce role separation, RL signals, or adversarial flow objectives to balance mode coverage and mode seeking (Jiang et al., 2025; Wu et al., 2026; Cheng et al., 2025; Lin et al., 2026). This complementarity is especially explicit in recent long-video work: Cai et al. (2026) pair a supervised global flow-matching head for long-range structure with a local DMD head for short-window fidelity, while HiAR (Zou et al., 2026) observes that self-rollout reverse-KL distillation can amplify low-motion shortcuts and adds a forward-KL regularizer to preserve motion diversity.
Video and autoregressive diffusion distillation.
Video distillation must handle temporal consistency and long-horizon error accumulation in addition to per-frame visual quality. Self-Forcing (Huang et al., 2025a) and APT2 (Lin et al., 2025b) are representative works that propose self-forcing as an on-policy distillation paradigm for mitigating exposure bias in AR generation. In particular, APT2 initializes self-forcing with teacher-forcing consistency distillation, but relies on a relatively cumbersome GAN objective during the self-forcing stage. Concurrent to our work, Causal Forcing++ (Zhao et al., 2026) also combines teacher-forcing consistency with self-forcing DMD, while we implement JVP-based continuous-time consistency under teacher forcing, and provide a systematic algorithm-and-infrastructure open recipe with holistic evaluation. Apart from the CM route, Transition Matching Distillation matches multi-step video denoising trajectories with few-step transition processes using conditional flow heads, followed by distribution matching on flow-head rollouts (Nie et al., 2026b). AnyFlow shifts video distillation from endpoint consistency to arbitrary-interval flow-map transitions and uses backward simulation for on-policy distillation in both bidirectional and causal architectures (Gu et al., 2026). Other recent work studies from-scratch few-step video training with efficient solution-flow objectives (Park et al., 2026), video-specific distillation losses for oversaturation and temporal collapse (You et al., 2026). Adversarial refinement has also been explored for one-step AR video generation, e.g., by augmenting DMD with a noised-latent GAN loss (Feng et al., 2026) or by using asymmetric adversarial distillation after distribution-matching warm-up (Li et al., 2026a). The distilled models are orthogonal to attention-level acceleration and could be further combined with sparse attention techniques (Zhang et al., 2025a, 2026), as demonstrated by TurboDiffusion (Zhang et al., 2025b), which combines rCM with attention acceleration and quantization.
6 Limitations and Future
Limitations.
Although Causal-rCM provides an effective algorithm-infrastructure recipe for autoregressive diffusion distillation, several limitations remain. First, frame-wise T2V training with long rollout depth is still fragile. In this setting, the 4-step SF-DMD model tends to develop camera drift after extended training, e.g., a consistent directional camera bias, and therefore cannot be trained for a long duration. This issue could be eliminated in action-conditioned interactive settings, where actions provide an explicit motion prior and reduce the ambiguity of camera evolution. Second, the best initialization before SF-DMD does not always translate into the best final model. TF-sCM gives a stronger pre-SF-DMD initialization than TF-dCM, but in the frame-wise setting, TF-dCM can be more stable under long SF-DMD refinement and achieve a higher final peak. This suggests that initialization quality and refinement stability are not fully aligned. Third, fully joint optimization like rCM remains challenging. In our causal setting, joint training tends to lower the VBench ceiling, so we currently use a staged pipeline. This could be attributed to the distribution gap between the causal teacher and the bidirectional teacher. Finally, the current custom-mask FlashAttention JVP kernel is implemented in Triton. As a result, the per-iteration speed of TF-sCM is only comparable to TF-dCM with standard FlashAttention-2, lacking behind more advanced kernels like FlashAttention-3/4.
Future directions.
A natural next step is to make the staged recipe more systematic. Table 6 summarizes our high-level view: current distillation methods can be interpreted as subsets of two ultimate pipelines, a CM route and a CTM route, each with bidirectional and causal variants. Discrete-time methods (dCM, MeanFlow with finite difference estimation) could be the warmup stage for continuous-time JVP ones (sCM, MeanFlow) to enhance stability.
| Route | Setting | Ultimate pipeline | Related works |
| CM | Bidirectional | dCM sCM DMD/GAN CM/on-policy CM | APT (Lin et al., 2025a): dCM GAN; rCM (Zheng et al., 2025d): sCM DMD. |
| CM | Causal | TF-dCM TF-sCM SF-DMD/SF-GAN TF-CM/SF-CM | APT2 (Lin et al., 2025b): TF-dCM SF-GAN; CF++ (Zhao et al., 2026): TF-dCM SF-DMD; Causal-rCM (ours): TF-dCM/TF-sCM SF-DMD. |
| CTM | Bidirectional | MeanFlow (FD) MeanFlow (JVP) DMD/GAN MeanFlow/on-policy MeanFlow | Transition Matching (Nie et al., 2026b): MeanFlow (FD) DMD2-v with flow-head rollout; AnyFlow (Gu et al., 2026): MeanFlow (FD) DMD + on-policy MeanFlow (FD). |
| CTM | Causal | TF-MeanFlow (FD) TF-MeanFlow (JVP) SF-DMD/SF-GAN TF-MeanFlow/SF-MeanFlow | AnyFlow (Gu et al., 2026): TF-MeanFlow (FD) SF-DMD + SF-MeanFlow (FD). |
Beyond algorithmic design, future work should improve the underlying systems stack. Better kernels for custom attention, JVP, and KV-cache execution, together with runtime features such as torch.compile, CUDA Graphs, and NVFP4 could further reduce overhead and make large-scale training and inference more efficient.
References
- Ali et al. (2025) Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, et al. World simulation with video foundation models for physical ai. arXiv preprint arXiv:2511.00062, 2025.
- Arriola et al. (2025) Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025.
- Bao et al. (2024) Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024.
- Brooks et al. (2024) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video-generation-models-as-world-simulators, 3, 2024.
- Cai et al. (2026) Shengqu Cai, Weili Nie, Chao Liu, Julius Berner, Lvmin Zhang, Nanye Ma, Hansheng Chen, Maneesh Agrawala, Leonidas Guibas, Gordon Wetzstein, et al. Mode seeking meets mean seeking for fast long video generation. arXiv preprint arXiv:2602.24289, 2026.
- Chen et al. (2024) Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024.
- Chen et al. (2025a) Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Junchen Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengcheng Ma, et al. Skyreels-v2: Infinite-length film generative model. arXiv preprint arXiv:2504.13074, 2025a.
- Chen et al. (2025b) Hansheng Chen, Kai Zhang, Hao Tan, Leonidas Guibas, Gordon Wetzstein, and Sai Bi. pi-flow: Policy-based few-step generation via imitation distillation. arXiv preprint arXiv:2510.14974, 2025b.
- Chen et al. (2026) Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, Ying-Cong Chen, Enze Xie, Xiaojuan Qi, and Song Han. Longlive2.0: An nvfp4 parallel infrastructure for long video generation. arXiv preprint arXiv, 2026.
- Cheng et al. (2025) Zhenglin Cheng, Peng Sun, Jianguo Li, and Tao Lin. Twinflow: Realizing one-step generation on large models with self-adversarial flows. arXiv preprint arXiv:2512.05150, 2025.
- Dong et al. (2024) Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2(3):4, 2024.
- Feng et al. (2026) Jiaqi Feng, Justin Cui, Yuanhao Ban, and Cho-Jui Hsieh. One-forcing: Towards stable one-step autoregressive video generation. arXiv preprint arXiv:2605.23458, 2026.
- Feng et al. (2025) Yao Feng, Chendong Xiang, Xinyi Mao, Hengkai Tan, Zuyue Zhang, Shuhe Huang, Kaiwen Zheng, Haitian Liu, Hang Su, and Jun Zhu. Vidarc: Embodied video diffusion model for closed-loop control. arXiv preprint arXiv:2512.17661, 2025.
- Gao et al. (2025) Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint arXiv:2506.09113, 2025.
- Geng et al. (2025) Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13447, 2025.
- Gu et al. (2026) Yuchao Gu, Guian Fang, Yuxin Jiang, Weijia Mao, Song Han, Han Cai, and Mike Zheng Shou. Anyflow: Any-step video diffusion model with on-policy flow map distillation. arXiv preprint arXiv:2605.13724, 2026.
- Hao-AI Lab (2026) Hao-AI Lab. FastVideo: A unified inference and post-training framework for accelerated video generation, 2026. URL https://github.com/hao-ai-lab/FastVideo.
- He et al. (2025) Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Size Wu, Wei Li, Xuchen Song, Yang Liu, Yangguang Li, and Yahui Zhou. Matrix-game 2.0: An open-source real-time and streaming interactive world model. arXiv preprint arXiv:2508.13009, 2025.
- Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
- Hong et al. (2025) Yicong Hong, Yiqun Mei, Chongjian Ge, Yiran Xu, Yang Zhou, Sai Bi, Yannick Hold-Geoffroy, Mike Roberts, Matthew Fisher, Eli Shechtman, Kalyan Sunkavalli, Feng Liu, Zhengqi Li, and Hao Tan. Relic: Interactive video world model with long-horizon memory. arXiv preprint arXiv:2512.04040, 2025.
- Hoogeboom et al. (2023) Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213–13232. PMLR, 2023.
- Huang et al. (2025a) Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009, 2025a.
- Huang et al. (2025b) Yubo Huang, Hailong Guo, Fangtai Wu, Weiqiang Wang, Shifeng Zhang, Shijie Huang, Qijun Gan, Lin Liu, Sirui Zhao, Enhong Chen, Jiaming Liu, and Steven Hoi. Live avatar: Streaming real-time audio-driven avatar generation with infinite length. arXiv preprint arXiv:2512.04677, 2025b.
- Huang et al. (2024) Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024.
- HunyuanWorld (2025) Team HunyuanWorld. Hy-world 1.5: A systematic framework for interactive world modeling with real-time latency and geometric consistency. arXiv preprint, 2025.
- Jacobs et al. (2023) Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023.
- Jiang et al. (2025) Dengyang Jiang, Dongyang Liu, Zanyi Wang, Qilong Wu, Liuzhuozheng Li, Hengzhuang Li, Xin Jin, David Liu, Changsheng Lu, Zhen Li, et al. Distribution matching distillation meets reinforcement learning. arXiv preprint arXiv:2511.13649, 2025.
- Jin et al. (2025) Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. In International Conference on Learning Representations, volume 2025, pages 23378–23402, 2025.
- Kim et al. (2023) Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
- Kim et al. (2026) Youngrae Kim, Qixin Hu, C-C Jay Kuo, and Peter A Beerel. Memrope: Training-free infinite video generation via evolving memory tokens. arXiv preprint arXiv:2603.12513, 2026.
- Kong et al. (2024) Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024.
- Li et al. (2026a) Haobo Li, Yanhong Zeng, Yunhong Lu, Jiapeng Zhu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Yujun Shen, and Zhipeng Zhang. Aad-1: Asymmetric adversarial distillation for one-step autoregressive video generation. arXiv preprint arXiv:2606.03972, 2026a.
- Li et al. (2026b) Haodong Li, Shaoteng Liu, Zhe Lin, and Manmohan Chandraker. Rolling sink: Bridging limited-horizon training and open-ended testing in autoregressive video diffusion. arXiv preprint arXiv:2602.07775, 2026b.
- Li et al. (2026c) Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998, 2026c.
- Lin et al. (2025a) Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 37959–37974. PMLR, 2025a.
- Lin et al. (2025b) Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, and Lu Jiang. Autoregressive adversarial post-training for real-time interactive video generation. arXiv preprint arXiv:2506.09350, 2025b.
- Lin et al. (2026) Shanchuan Lin, Ceyuan Yang, Zhijie Lin, Hao Chen, and Haoqi Fan. Continuous adversarial flow models. arXiv preprint arXiv:2604.11521, 2026.
- Lipman et al. (2022) Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
- Liu et al. (2026) Jinxiu Liu, Xuanming Liu, Kangfu Mei, Yandong Wen, Ming-Hsuan Yang, and Weiyang Liu. Streaming autoregressive video generation via diagonal distillation. In ICLR, 2026.
- Liu et al. (2022) Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022.
- Lu and Song (2024) Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.11081, 2024.
- Lu et al. (2022) Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International conference on machine learning, pages 14429–14460. PMLR, 2022.
- Luhman and Luhman (2021) Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
- Nie et al. (2026a) Weili Nie, Julius Berner, Chao Liu, and Arash Vahdat. Nvidia fastgen: Fast generation from diffusion models, 2026a. URL https://github.com/NVlabs/FastGen.
- Nie et al. (2026b) Weili Nie, Julius Berner, Nanye Ma, Chao Liu, Saining Xie, and Arash Vahdat. Transition matching distillation for fast video generation. arXiv preprint arXiv:2601.09881, 2026b.
- Ning et al. (2024) Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. In International Conference on Learning Representations, volume 2024, pages 15167–15189, 2024.
- NVIDIA (2026) NVIDIA. Cosmos 3: Omnimodal world models for physical ai. arXiv preprint arXiv:2606.02800, 2026. URL https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf.
- Park et al. (2026) Dogyun Park, Yanyu Li, Sergey Tulyakov, and Anil Kag. Eflow: Fast few-step video generator training from scratch via efficient solution flow. arXiv preprint arXiv:2603.27086, 2026.
- Peng et al. (2025) Yansong Peng, Kai Zhu, Yu Liu, Pingyu Wu, Hebei Li, Xiaoyan Sun, and Feng Wu. Facm: Flow-anchored consistency models. arXiv preprint arXiv:2507.03738, 2025.
- Pu et al. (2025) Yifan Pu, Yizeng Han, Zhiwei Tang, Jiasheng Tang, Fan Wang, Bohan Zhuang, and Gao Huang. Few-step distillation for text-to-image generation: A practical guide. arXiv preprint arXiv:2512.13006, 2025.
- Robbyant Team et al. (2026) Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, Yihang Chen, Jie Liu, Yansong Cheng, Yao Yao, Jiayi Zhu, Yihao Meng, Kecheng Zheng, Qingyan Bai, Jingye Chen, Zehong Shen, Yue Yu, Xing Zhu, Yujun Shen, and Hao Ouyang. Advancing open-source world models. arXiv preprint arXiv:2601.20540, 2026.
- Sabour et al. (2025) Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your flow: Scaling continuous-time flow map distillation. arXiv preprint arXiv:2506.14603, 2025.
- Sahoo et al. (2024) Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524, 2024.
- Schmidt (2019) Florian Schmidt. Generalization in generation: A closer look at exposure bias. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 157–167, 2019.
- Seedance et al. (2026) Team Seedance, De Chen, Liyang Chen, Xin Chen, Ying Chen, Zhuo Chen, Zhuowei Chen, Feng Cheng, Tianheng Cheng, Yufeng Cheng, et al. Seedance 2.0: Advancing video generation for world complexity. arXiv preprint arXiv:2604.14148, 2026.
- Shi et al. (2024) Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024.
- Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
- Song et al. (2023) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 32211–32252. PMLR, 2023.
- Teng et al. (2025) Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, et al. Magi-1: Autoregressive video generation at scale. arXiv preprint arXiv:2505.13211, 2025.
- Tong et al. (2025) Shangyuan Tong, Nanye Ma, Saining Xie, and Tommi Jaakkola. Flow map distillation without data. arXiv preprint arXiv:2511.19428, 2025.
- Wan et al. (2025) Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
- Wang et al. (2023) Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in neural information processing systems, 36:8406–8441, 2023.
- Wu et al. (2026) Tianhe Wu, Ruibin Li, Lei Zhang, and Kede Ma. Diversity-preserved distribution matching distillation for fast visual synthesis. arXiv preprint arXiv:2602.03139, 2026.
- Yang et al. (2026) Shuai Yang, Wei Huang, Ruihang Chu, Yicheng Xiao, Yuyang Zhao, Xianbang Wang, Muyang Li, Enze Xie, Yingcong Chen, Yao Lu, et al. Longlive: Real-time interactive long video generation. In ICLR, 2026.
- Ye et al. (2025) Haotian Ye, Kaiwen Zheng, Jiashu Xu, Puheng Li, Huayu Chen, Jiaqi Han, Sheng Liu, Qinsheng Zhang, Hanzi Mao, Zekun Hao, et al. Data-regularized reinforcement learning for diffusion models at scale. arXiv preprint arXiv:2512.04332, 2025.
- Ye et al. (2026) Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026.
- Yesiltepe et al. (2026) Hidir Yesiltepe, Tuna Meral, Adil Kaan Akan, Kaan Oktay, and Pinar Yanardag. Infinity-rope: Action-controllable infinite video generation emerges from autoregressive self-rollout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 40256–40265, 2026.
- Yi et al. (2025) Jung Yi, Wooseok Jang, Paul Hyunbin Cho, Jisu Nam, Heeji Yoon, and Seungryong Kim. Deep forcing: Training-free long video generation with deep sink and participative compression. arXiv preprint arXiv:2512.05081, 2025.
- Yin et al. (2024a) Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455–47487, 2024a.
- Yin et al. (2024b) Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6613–6623, 2024b.
- Yin et al. (2025) Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22963–22974, 2025.
- You et al. (2026) Yuyang You, Yongzhi Li, Jiahui Li, Yadong Mu, Quan Chen, and Peng Jiang. Adaptive video distillation: Mitigating oversaturation and temporal collapse in few-step generation. arXiv preprint arXiv:2603.21864, 2026.
- Zewei and Yunpeng (2025) Tao Zewei and Huang Yunpeng. Magiattention: A distributed attention towards linear scalability for ultra-long context, heterogeneous mask training. https://github.com/SandAI-org/MagiAttention/, 2025.
- Zhang et al. (2025a) Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, et al. Sla: Beyond sparsity in diffusion transformers via fine-tunable sparse-linear attention. arXiv preprint arXiv:2509.24006, 2025a.
- Zhang et al. (2025b) Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. Turbodiffusion: Accelerating video diffusion models by 100-200 times. arXiv preprint arXiv:2512.16093, 2025b.
- Zhang et al. (2026) Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang, Ion Stoica, Jianfei Chen, Jun Zhu, and Joseph E Gonzalez. Sla2: Sparse-linear attention with learnable routing and qat. arXiv preprint arXiv:2602.12675, 2026.
- Zhao et al. (2026) Min Zhao, Hongzhou Zhu, Kaiwen Zheng, Zihan Zhou, Bokai Yan, Xinyuan Li, Xiao Yang, Chongxuan Li, and Jun Zhu. Causal forcing++: Scalable few-step autoregressive diffusion distillation for real-time interactive video generation. arXiv preprint arXiv:2605.15141, 2026.
- Zhao et al. (2023) Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023.
- Zheng et al. (2023a) Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems, 36:55502–55542, 2023a.
- Zheng et al. (2023b) Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pages 42363–42389. PMLR, 2023b.
- Zheng et al. (2025a) Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. Diffusionnft: Online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117, 2025a.
- Zheng et al. (2025b) Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Direct discriminative optimization: Your likelihood-based visual generative model is secretly a gan discriminator. arXiv preprint arXiv:2503.01103, 2025b.
- Zheng et al. (2025c) Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In International Conference on Learning Representations, volume 2025, pages 63186–63227, 2025c.
- Zheng et al. (2025d) Kaiwen Zheng, Yuji Wang, Qianli Ma, Huayu Chen, Jintao Zhang, Yogesh Balaji, Jianfei Chen, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Large scale diffusion distillation via score-regularized continuous-time consistency. arXiv preprint arXiv:2510.08431, 2025d.
- Zhou et al. (2024) Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
- Zhu et al. (2026) Hongzhou Zhu, Min Zhao, Guande He, Hang Su, Chongxuan Li, and Jun Zhu. Causal forcing: Autoregressive diffusion distillation done right for high-quality real-time interactive video generation. arXiv preprint arXiv:2602.02214, 2026.
- Zou et al. (2026) Kai Zou, Dian Zheng, Hongbo Liu, Tiankai Hang, Bin Liu, and Nenghai Yu. Hiar: Efficient autoregressive long video generation via hierarchical denoising. arXiv preprint arXiv:2603.08703, 2026.
Appendix A Theoretical Analysis of TrigFlow-sCM and RF-sCM
This section compares two implementations of continuous-time consistency distillation for an RF-native velocity predictor: (i) applying a TrigFlow wrapper to the RF velocity predictor and then using the TrigFlow-sCM objective (Zheng et al., 2025d), and (ii) directly writing the sCM objective under the RF schedule. Despite that different diffusion noise schedules (e.g., TrigFlow and RF) are equivalent and mutually convertible (Zheng et al., 2023b) up to a scaling factor, we show that they result in generally different normalized MSE training objectives for sCM. The difference comes from the input-output scaling of the TrigFlow wrapper, the tangent normalization, and the finite-precision evaluation order of JVPs.
RF and TrigFlow coordinates.
Let denote the RF time and let denote the TrigFlow time. Define
The RF and TrigFlow forward processes are
They are related by a time-dependent state scaling:
| (21) |
Let be an RF velocity predictor and let denote the RF teacher velocity. The direct RF consistency map is
| (22) |
TrigFlow wrapper as input-output transforms.
The TrigFlow wrapper around the RF velocity predictor can be written as
| (23) |
Equivalently, the wrapper first applies the input transform , evaluates the RF velocity predictor, and then applies the output transform
Under the TrigFlow preconditioning
| (24) |
substituting Eqn. 23 gives
| (25) |
Therefore, the TrigFlow wrapper and the direct RF parameterization define the same consistency map after the change of variables in Eqn. 21.
Direct RF-sCM tangent.
The RF teacher ODE is
For the stop-gradient network , define the RF JVP
| (26) |
The tangent of the RF consistency map is
| (27) |
The direct RF-sCM objective can thus be written as
| (28) |
is zero in the forward value, but it still indicates the output coordinate with respect to which gradients are taken.
TrigFlow JVP through the input and output transforms.
We next rewrite the TrigFlow-sCM JVP in the RF coordinates. Along the TrigFlow teacher trajectory,
| (29) |
Using and , where
we obtain
| (30) |
Since
the explicit state terms cancel and
| (31) |
Thus the JVP direction entering the RF velocity predictor inside the TrigFlow wrapper is
| (32) |
In exact arithmetic,
| (33) |
The output transform in Eqn. 23 contains explicit -dependent coefficients. Therefore the JVP of the wrapped velocity is
| (34) |
Now differentiate the TrigFlow consistency map in Eqn. 24:
| (35) |
Substituting ,
and Eqn. 34, we get
| (36) |
Using
we obtain the compact relation
| (37) |
This derivation makes explicit that the TrigFlow wrapper introduces no new RF-JVP structure: after the input and output transforms, the same RF combination
appears. The only exact-arithmetic difference at the tangent level is the factor .
TrigFlow-sCM objective in RF velocity coordinates.
The TrigFlow-sCM objective is applied in the TrigFlow velocity coordinate. From Eqn. 23,
| (38) |
Therefore, with
where is taken as in sCM (Lu and Song, 2024) and rCM (Zheng et al., 2025d), the TrigFlow-sCM loss becomes
| (39) |
By contrast, direct RF-sCM is Eqn. 28. Hence the two objectives share the same zero-consistency condition,
but they are not, in general, the same normalized MSE objective.
Effect of tangent normalization.
The distinction is easiest to see when . Eqn. 28 gives the RF normalized tangent target
Eqn. 39 gives the TrigFlow target expressed in the RF output coordinate:
Thus, if is kept general,
The loss gradient with respect to the RF velocity predictor satisfies
| (40) |
When , the difference is not reducible to a simple scalar reweighting, because the TrigFlow denominator becomes , whereas the RF denominator is . Therefore, tangent normalization breaks the strict equivalence of the two normalized MSE losses.
Finite-precision JVP evaluation.
The relation in Eqn. 33 is exact only in real arithmetic. In floating-point computation,
| (41) |
in general, and the JVP rearrangement (Lu and Song, 2024) further absorbs the coefficient into JVP computation. Placing the scales inside the JVP direction propagates the scaled tangent through every layer of the network, while factoring them outside first evaluates an unscaled tangent and only then rescales the result. These two evaluation orders can differ because of rounding, mixed-precision casts, fused kernels, activation checkpointing, and custom FlashAttention JVP implementations.
Moreover, the TrigFlow wrapper contains explicit input-output transform terms whose cancellations are algebraically exact but not necessarily bitwise exact. For example, the derivation of Eqn. 37 cancels the state-dependent terms from differentiating , , and the wrapper coefficients. A direct RF implementation computes the compact expression
without these intermediate transform terms. Consequently, even when the exact-arithmetic tangent relation holds, the two implementations are not expected to be bitwise equivalent under practical large-scale mixed-precision training. This numerical distinction can be amplified by the normalized target , especially when is small or the stabilizing constant is small.
Appendix B FlashAttention-2 JVP Kernel with Custom Masks
For TF-sCM, the student network is evaluated on a packed sequence that concatenates clean context tokens and noisy target tokens under a TF attention mask. The Jacobian-vector-product (JVP) must be computed through exactly the same masked attention operator as the primal forward pass. A dense additive mask is conceptually simple but memory-inefficient for long video sequences. We therefore represent the custom mask as a sparse list of admissible query-key rectangles in the MagiAttention (Zewei and Yunpeng, 2025) style, and stream only those rectangles inside the FlashAttention-2 loop.
Let be a custom attention mask, where means that query token may attend to key token , and otherwise. The masked attention output is
| (42) |
For JVP, given tangents , the score tangent is
| (43) |
The mask is a discrete routing object and has no tangent. Therefore, masked-out entries are assigned zero tangent contribution:
Equivalently, the JVP is taken through the masked attention map
with fixed.
For a row , let denote the masked softmax probability over allowed keys. The attention tangent is
| (44) |
where all sums are over valid keys under . The kernel computes this expression in the same online-softmax pass as the primal FlashAttention computation. For a streamed block, define the unnormalized probability
Besides the standard FlashAttention accumulators , we maintain three JVP accumulators:
After normalization, the tangent output is
| (45) |
where in the last term is the normalized primal output. The same online rescaling factor used for the primal accumulators is applied to , so the JVP remains numerically aligned with the FlashAttention-2 softmax normalization.
Sparse custom-mask representation.
The custom mask is represented as a set of query groups and their admissible key ranges. Each query group contains a contiguous query interval and a list of valid key intervals
The kernel launches tasks , where is a query tile inside . Each task streams only the key ranges in . This range-list view covers both dense/full attention and structured causal masks: dense attention has one query group with one full key range, while teacher-forcing or block-causal masks are decomposed into a small number of full query-key rectangles. Importantly, the same sparse schedule is used for both the primal score and its tangent , ensuring that the tangent corresponds to the exact masked attention operator used in the forward pass.
We present the full algorithm in Algo. 1.