OmniCap-IF:全能视频描述的指令遵循基准与模型改进
阅读原文· arxiv.orgOmniCap-IF 是首个针对全能模态大语言模型(OLLMs)视频描述指令遵循能力的基准,涵盖纯视觉、纯音频和视听三种模态下的 50 种约束类型,并引入时间定位评估时空精确性。在 1920 个高质量样本上的评测显示模型间存在显著性能差距,并发现“格式-内容权衡”——格式复杂度增加会损害模型的全能模态推理能力。研究团队还构建了 54K 指令微调数据集 OmniCap-IF-54K,并发布 OmniCaptioner-IF 模型,在复杂指令遵循与通用全能模态描述性能上均取得明显提升。
While Omni-modal Large Language Models (OLLMs) have demonstrated impressive capabilities in jointly processing audio and visual streams, their ability to strictly adhere to complex, multi-faceted user instructions remains largely unexplored. Existing benchmarks primarily focus on holistic video understanding or text-only instruction following, failing to capture the intricate interplay between modalities and user constraints. To bridge this gap, we introduce OmniCap-IF, the first comprehensive benchmark specifically designed to evaluate instruction-following capabilities in omni-modal captioning. OmniCap-IF incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness. Our benchmark encompasses 50 distinct constraint types across pure visual, pure audio, and audio-visual modalities, while integrating Temporal Grounding to assess spatio-temporal precision. Extensive evaluations of prominent models on 1,920 high-quality samples reveal significant performance disparities. Furthermore, our analysis uncovers a critical "format-content tradeoff", demonstrating that increasing formatting complexity directly degrades models' omni-modal reasoning abilities. Finally, to advance the field, we curate a 54K instruction-tuning dataset, OmniCap-IF-54K and present OmniCaptioner-IF, which achieves notable improvements in both complex instruction adherence and general omni-modal captioning performance.