# OmniVideo-100K：通过结构化脚本和证据链进行视听推理的数据集

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-12 08:00
- AIHOT 分数：43
- AIHOT 链接：https://aihot.virxact.com/items/cmqeknks0034uslune0hwsckk
- 原文链接：https://arxiv.org/abs/2606.14702

## AI 摘要

针对视频-音频问答中跨模态关联薄弱、长程时序连接不足的问题，提出自动数据引擎，包含实体锚定视频脚本化（生成摘要、主实体列表和片段描述）和线索引导QA生成两个机制。基于该流程构建指令微调数据集OmniVideo-100K及人工测试集OmniVideo-Test。在VITA-1.5、Qwen2.5-Omni-7B和Qwen3-Omni-30B上微调后，OmniVideo-Test性能最高提升20.59%，在Daily-Omni、JointAVBench等基准上最多提升12.64%。

## 正文

Current automated pipelines for audio-visual Question Answering (QA) generally adopt a ``video-caption-QA'' paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) Entity-Anchored Video Scripting transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) Clue-Guided QA Generation prompts models to first mine cross-segment, multimodal clues from the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct the instruction-tuning dataset OmniVideo-100K and a human-verified test set, OmniVideo-Test. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B on OmniVideo-100K yields performance gains of up to 20.59% on OmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks like Daily-Omni and JointAVBench.