GEM:生成式监督助力具身智能
阅读原文· arxiv.org当前具身视觉语言模型(VLM)在高层语义与底层物理操作能力间存在差距。GEM 模型通过在预训练阶段集成深度图生成这一生成式监督任务来弥补该差距,联合训练显著提升了模型的语义理解与物理操作能力。配套发布的 GEM-4M 大规模数据集包含定位、推理和规划数据,并附带高质量深度监督。实验表明,GEM 在多项具身基准测试中达到领先水平,其部署的行动模型 GEM-VLA 在仿真与真实环境评估中均展现出卓越的任务执行能力。代码、模型与数据集已开源。
Embodied Vision-Language Models (VLMs) have demonstrated impressive performance and generalization in robotics, particularly within Vision-Language-Action frameworks. However, a significant gap remains between the high-level semantic focus of standard text-guided pre-training paradigms and the low-level spatial and physical knowledge critical for execution in embodied environments. In this paper, we introduce GEM, a Generative-supervised Embodied vision-language Model designed to bridge this divide. We propose integrating a depth map generation task directly into the VLM pre-training phase. By training this generative objective jointly with the main model, we observe substantial improvements in embodied intelligence, significantly enhancing both semantic understanding and physical operation capabilities. To support this paradigm, we curate and release GEM-4M, a comprehensive large-scale dataset featuring a mixture of grounding, reasoning, and planning data paired with high-quality depth supervision. Extensive experiments demonstrate that GEM achieves state-of-the-art results across diverse embodied benchmarks. Furthermore, our deployed action model, GEM-VLA, exhibits vastly superior task execution abilities in both simulation environments and real-world evaluations. Code, models, and datasets are available at https://zhaorw02.github.io/GEM/