GeoVR:从视频学习几何表示,重塑多模态大语言模型的空间智能
阅读原文· arxiv.org多模态大语言模型(MLLM)擅长2D语义理解,但缺乏3D空间一致性。GeoVR框架利用纯2D视频序列,通过从预训练3D基础模型蒸馏几何知识,重构MLLM的语义隐空间。其多目标学习策略包含四个互补几何约束:帧间相机位姿估计、密集深度图回归、度量尺度因子预测以及多尺度3D特征对齐。在空间推理基准上,GeoVR达到当前最佳性能(SOTA),为赋予基础模型空间智能提供了新范式。
Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames. Given the scarcity of large-scale 3D data, we present GeoVR, a novel framework that learns geometric representations using purely 2D video sequences. This approach effectively restructures the semantic latent space within MLLMs to unlock spatial intelligence. Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge from pre-trained 3D foundation models. This is accomplished through a multi-objective learning strategy driven by four complementary geometric targets: (1) estimating inter-frame camera poses to embed varying viewpoint dynamics, (2) regressing dense depth maps to anchor physical distances, (3) predicting a metric scale factor for real-world calibration, and (4) distilling multi-scale 3D features to align the intermediate feature space. Guided by these explicit physical and geometric constraints, the model's internal representations naturally develop strong 3D awareness. Extensive experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance, establishing a new paradigm for endowing foundation models with spatial intelligence.