LVSA:长视频扩散模型的免训练稀疏注意力机制
阅读原文· arxiv.orgLVSA是一种无需训练、适用于视频扩散Transformer的块稀疏注意力机制,旨在降低长视频推理的计算成本。它通过结合结构化窗口模式与旋转全局锚点,避免了导致时程伪影的固定网格偏置。该技术在Wan 2.1 1.3B、Wan 2.1 14B及HunyuanVideo 1.5上分别实现了最高3.17倍、2.98倍和3.33倍的计算量缩减,并使HunyuanVideo 1.5在单一GPU上能生成2倍于训练长度的视频。LVSA在NPU上同样有效。此外,论文还提出了VQeval评估工具,专门针对循环视频故障进行评分,以弥补现有评估工具的不足。
Dense self-attention is the compute and quality bottleneck of long-video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long Video Sparse Attention (LVSA), a training-free model-agnostic block-sparse attention for video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.