LoomVideo:高效统一视频生成与编辑架构
阅读原文· arxiv.orgLoomVideo是一个5B参数的高效统一架构,支持视频生成与编辑。它用多模态大语言模型(MLLM)替换标准文本编码器,通过Deepstack注入机制对齐MLLM特征与扩散Transformer。核心创新为零开销的Scale-and-Add条件方法,直接缩放和加性处理源视频潜变量,避免token拼接导致的序列长度翻倍与自注意力计算复杂度增加。集成Negative Temporal RoPE策略处理多张参考图像。该模型在综合基准上达到或接近SOTA,尤其在电商和时尚生成场景表现突出,推理速度相比同类模型提升至少5.41倍。
Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier field. Existing unified frameworks predominantly rely on massive models (typically 13B parameters or more) and incorporate source video conditions for editing by concatenating sequence tokens. This concatenation inevitably doubles the sequence length, quadrupling the computational complexity of the self-attention mechanism and introducing prohibitive overhead. To address these bottlenecks, we present LoomVideo, a highly efficient 5B-parameter unified architecture for both video generation and editing. LoomVideo replaces the standard text encoder with a Multimodal Large Language Model (MLLM) and employs Deepstack injection mechanism to align multi-layer MLLM features with the Diffusion Transformer (DiT). Crucially, we introduce a zero-overhead Scale-and-Add conditioning approach for video editing. By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation, drastically reducing computational cost while maintaining robust capabilities for complex, non-rigid edits. Furthermore, a Negative Temporal RoPE strategy is seamlessly integrated to handle multiple reference images. Extensive experiments demonstrate that our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, exhibiting exceptional superiority in e-commerce and fashion generation scenarios. Benefiting from the zero-overhead conditioning mechanism, LoomVideo achieves at least a 5.41x acceleration in inference speed compared to models of similar capabilities, paving the way for highly practical and efficient video foundation models.