Lip Forcing:用于实时唇同步的少步自回归扩散方法
阅读原文· arxiv.orgLip Forcing提出了自回归扩散方法用于视频到视频唇同步,从14B参数的音频条件双向视频扩散教师模型蒸馏出因果学生模型。推理时每个块仅需两步去噪,无需CFG,实现实时流式处理。技术分析揭示CFG的保真度-同步权衡,进而衍生出Sync-Window DMD、两步推理调度和基于SyncNet的奖励三项组件。1.3B学生模型在31 FPS下实时输出,比同规模双向模型快17.6倍;14B学生模型是目前最大的V2V唇同步扩散模型,比教师快39.8倍,保真度接近。首帧时延均小于1毫秒。
Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, 17.6times faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs 39.8times faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.