Video-MME-Logical:视频时间逻辑推理的受控诊断基准
阅读原文· arxiv.orgVideo-MME-Logical围绕五种时间逻辑操作(状态跟踪、顺序计数、时序排序、动态空间性、结构组合)构建,包含25个细粒度任务类别,通过控制对象状态、转换和逻辑组合来分离评估多模态大语言模型(MLLM)的视频时间逻辑推理能力。实验表明,当前SOTA MLLM与人类之间存在显著差距,且随时间逻辑复杂度增加而扩大。即使对多达500K生成样本进行监督微调,仍无法弥合这一推理鸿沟。该基准为分析和改进MLLM的时间逻辑推理提供了可扩展的测试平台。
Recent interest in multimodal large language models (MLLMs) raises a central question: can they reason over dynamic visual evidence rather than merely recognize objects or events in individual frames? This ability, which we refer to as video temporal-logical reasoning, requires models to maintain, update, and compose evidence as visual states evolve across frames. Existing video benchmarks often conflate this capability with scene complexity, static recognition, or uncontrolled temporal variation. To isolate this capability, we introduce Video-MME-Logical, a controlled benchmark organized around five temporal-logical operations: state tracking, sequential counting, temporal ordering, dynamic spatiality, and structural composition. The benchmark contains 25 fine-grained task categories generated with controlled object states, transitions, temporal dependencies, and logical compositions. It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity, and supports intermediate-state diagnostics by verifying whether models recover the required logical reasoning trace before producing the final answer. Experiments with state-of-the-art MLLMs reveal a substantial human-model gap, especially as temporal-logical complexity increases. Supervised fine-tuning on up to 500K generated samples improves performance but remains insufficient to close the reasoning gap, positioning Video-MME-Logical as a scalable testbed for analyzing and improving temporal-logical reasoning in MLLMs.