CogOmniControl:基于创意意图认知的推理驱动可控视频生成
阅读原文· arxiv.org针对现有视频生成模型在抽象或复杂控制下表现不佳的问题,CogOmniControl框架将任务解耦为创意意图认知与内容生成。该研究通过动画数据训练专用CogVLM,提升对模糊用户意图的专业理解与推理能力。同时,利用CogOmniDiT模块统一多种条件控制,并通过强化学习将其输出与CogVLM推理对齐。系统还基于专业工作流数据构建了CogReasonBench和CogControlBench基准。实验证明,CogOmniControl性能超越现有开源模型。
Recent diffusion models achieve strong photorealism and fluency in video generation, yet remain fragile under abstract, sparse or complex conditions, leading to poor performance in professional production workflows such as storyboard sketches and clay render conditions. Existing video generation models, either inject conditions through adapters or couple a generic vision-language model (VLM) within a diffusion backbone, leaving a capability gap and failing to produce the videos that align with the user's creative intent. We present CogOmniControl, a reasoning-driven framework that factorizes controllable video generation into creative intent cognition and generation. Specifically, we train a specialized CogVLM using authentic anime production data. Compared to generic VLMs, it generates more professional and clear outputs, accurately cognizing user creative intent from sparse and abstract conditions and tuning these cues into dense reasoning output. Besides, CogOmniDiT unifies the controls from various conditions through in-context generation and is aligned to the CogVLM reasoning outputs via reinforcement learning. Furthermore, leveraging CogVLM's robust capability in guiding video generation, we release its potential in planning specific evaluators and enable a Best-of-N selection for the generated videos. This integration transforms the entire framework into a closed-loop "harness-like" architecture. We further introduce CogReasonBench and CogControlBench, built from professional workflows data that carry genuine creative intent rather than simulated ones. Experiments on two benchmarks show that CogOmniControl surpassed the existing open-source models. The project website: https://um-lab.github.io/CogOmniControl/