We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.
SemComp-Bench:为视频生成中的语义任务完成设立基准
AI 导读
研究团队提出“语义任务完成视频生成”这一以结果为导向的视频生成任务,要求模型同时实现预期结果并保持与参考图像的语义对齐。为此构建了覆盖六个领域的评估数据集 SemComp-Data,并推出基于视觉语言模型(VLM)的评估协议 SemComp-Bench,分别以 OA Score 和 GR Score 衡量结果达成与生成可靠性。实验显示,现有代表性视频生成模型在兼顾任务目标与语义对齐方面仍面临挑战。
HuggingFace Daily Papers(社区热门论文)
43
AI 编辑部评分,满分 100SemComp-Bench:为视频生成中的语义任务完成设立基准
研究团队提出“语义任务完成视频生成”这一以结果为导向的视频生成任务,要求模型同时实现预期结果并保持与参考图像的语义对齐。为此构建了覆盖六个领域的评估数据集 SemComp-Data,并推出基于视觉语言模型(VLM)的评估协议 SemComp-Bench,分别以 OA Score 和 GR Score 衡量结果达成与生成可靠性。实验显示,现有代表性视频生成模型在兼顾任务目标与语义对齐方面仍面临挑战。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org