DomainShuttle:面向开放域主题驱动的文本到视频生成
阅读原文· arxiv.orgDomainShuttle 提出一种面向开放域主题驱动文本到视频生成(S2V)的方法,支持域内(高保真保留参考主体特征)和跨域(允许主体无关属性随文本提示灵活变化)两种场景。该方法引入 Domain-MoT 模块,通过解耦视频与参考特征并采用域感知 AdaLN 进行主体特定建模;提出 Video-Reference DualRoPE 方案,将参考图像 token 与视频 token 置于独立 RoPE 空间实现主体级空间建模;设计 Cross-Pair Consistent Loss 提取不受无关特征干扰的内在主体特征。实验表明,DomainShuttle 在多种开放域场景中相比现有方法实现显著提升,兼具高主体保真度与生成灵活性。
Open domain subject-driven text-to-video (S2V) generation has drawn significant interest in academia and industry. Open domain S2V mainly involves two scenarios: in-domain, which requires retaining the reference subject features as much as possible, and cross-domain, which preserves the intrinsic features of the subject while allowing subject-irrelevant properties to vary flexibly according to the text prompt. Existing methods primarily focus on maximizing subject fidelity in in-domain scenarios, which limits their editability and adaptability in cross-domain scenarios, such as novel styles, semantic combinations, or domain attributes. In this study, we propose that an ideal S2V method should flexibly shuttle between different domains, achieving strong performance in both in-domain and cross-domain scenarios. To this end, we propose DomainShuttle, which could achieve high fidelity and generative flexibility for open domain video personalization. Specifically, we introduce Domain-MoT, which decouples videos and reference features and introduces the domain-aware AdaLN for domain-specific modeling of reference images. We then introduce the Video-Reference DualRoPE scheme, which places reference image tokens and video tokens in separate RoPE spaces to enable precise subject-level spatial modeling, and Cross-Pair Consistent Loss, which aims to extract intrinsic subject features unaffected by irrelevant features. Extensive experiments demonstrate that DomainShuttle achieves significant performance improvements over existing methods, exhibiting high subject fidelity and generative flexibility across diverse open domain application scenarios.