Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and…
Beyond Visual CoT:Internalized Visual Thinking 实现主动视频推理
AI 导读
多模态大语言模型常用视觉思维链(Visual CoT)进行空间、时间与具身环境推理,但生成中间推理图像带来大量推理开销。新提出的后训练框架 Internalized Visual Thinking(IVT)在训练阶段内化视觉思考,推理时直接进行文本预测与优化,从而在不增加推理成本的前提下实现主动视频推理。
Apple Machine Learning Research(RSS)
50
AI 编辑部评分,满分 100Beyond Visual CoT:Internalized Visual Thinking 实现主动视频推理
多模态大语言模型常用视觉思维链(Visual CoT)进行空间、时间与具身环境推理,但生成中间推理图像带来大量推理开销。新提出的后训练框架 Internalized Visual Thinking(IVT)在训练阶段内化视觉思考,推理时直接进行文本预测与优化,从而在不增加推理成本的前提下实现主动视频推理。
原文 · 保持原样,未翻译原文 · 未翻译
来源:Apple Machine Learning Research(RSS)· machinelearning.apple.com