Hybrid-thinking multimodal large language models (MLLMs) allow a single model to alternate between deliberative thinking and latency-efficient non-thinking inference. Although these modes differ in reasoning budget, their delivered responses should satisfy the same user-facing standard. Correctness alone may not characterize this response quality; we therefore evaluate task accuracy and response-pattern failures as complementary outcomes. We study this gap through response-pattern alignment: whether thinking and non-thinking interfaces preserve acceptable final-response behavior. We introduce PatternEval, a failure-enriched diagnostic benchmark comprising 2,415 multimodal prompts spanning visual perception and grounding, structured image understanding, and multimodal knowledge reasoning. PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Response-pattern failures are widespread across models from different providers, with non-thinking inference exhibiting substantially higher failure rates and thereby creating systematic misalignment between thinking and non-thinking interfaces. Motivated by this diagnosis, we develop PatternRM, a response-level reward model, and PatternRL, which introduces pattern-specific penalties during reinforcement learning. Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that incorporating pattern-specific penalties into reinforcement learning can mitigate cross-mode misalignment while incurring a marginal task performance trade-off. Together, PatternEval and PatternRL provide an evaluation-and-training framework for aligning user-visible response patterns across hybrid-thinking interfaces.
超越正确率:混合思考多模态大模型的响应行为基准与对齐
AI 导读
混合思考多模态大模型(MLLM)可在深思推理与低延迟非思考推理间切换,但两种模式输出的响应行为未必一致。新研究推出PatternEval基准,含2,415个多模态提示,检测思维链泄露、响应重复、逻辑矛盾与表演性推理四类失败,发现非思考推理失败率显著更高。配套的PatternRM与PatternRL在Qwen3-VL-4B和Qwen3-VL-8B上验证,可缓解跨模式错位且任务性能损失轻微。
HuggingFace Daily Papers(社区热门论文)
52
AI 编辑部评分,满分 100超越正确率:混合思考多模态大模型的响应行为基准与对齐
混合思考多模态大模型(MLLM)可在深思推理与低延迟非思考推理间切换,但两种模式输出的响应行为未必一致。新研究推出PatternEval基准,含2,415个多模态提示,检测思维链泄露、响应重复、逻辑矛盾与表演性推理四类失败,发现非思考推理失败率显著更高。配套的PatternRM与PatternRL在Qwen3-VL-4B和Qwen3-VL-8B上验证,可缓解跨模式错位且任务性能损失轻微。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org