Skill-3D:面向智能体3D空间推理的场景感知技能自演进框架
阅读原文· arxiv.orgSkill-3D框架通过场景记忆与技能库协同演化,解决MLLM智能体在3D空间推理中工具使用偏好固化的问题。框架记录智能体的工具使用轨迹,将同类场景的成功轨迹聚合蒸馏成可复用技能,失败轨迹作为教训附于技能。当类似场景再现时,注入对应技能指导智能体,新轨迹反向优化技能,形成记忆与技能库自演进循环。实验表明,该方法在VSI-Bench上将工具利用率从39%提升至78%;在MMSI-Bench上使Gemini-3-Flash提升67%;对Qwen3-VL-8B进行技能轨迹后训练后,在VSI-Bench上提升43%。
This paper explores agentic 3D spatial understanding, i.e., MLLM agents performing 3D reasoning through tool use. Existing methods often misuse tools and exhibit biased tool preferences under 3D scenarios, leaving the agentic paradigm with only marginal gains over non-agentic strategies. We reveal that 3D spatial reasoning tasks are heterogeneous across scenes, while these agents apply a uniform tool-use strategy to all scenes rather than selecting tools according to the specific scene and task. To address this, we propose Skill-3D, a framework that learns self-evolving scene-aware skills. Specifically, Skill-3D identifies the task scene and records the agent's tool-use trajectory into a Scene Memory, where successful trajectories from similar scenes are aggregated and distilled into a reusable scene-aware skill, with failed ones attached to the skill as lessons. During training, once a similar scene recurs, the corresponding skill is injected to guide the agent, producing new trajectories whose successes and failures further refine the skill, forming a loop in which the memory and the skill library co-evolve. Experiments show that Skill-3D substantially improves tool utilization in 3D spatial reasoning (from 39% to 78% on VSI-Bench), driving the agent toward correct and sufficient tool use. For instance, it improves Gemini-3-Flash by 67% on MMSI-Bench. Furthermore, we conduct agentic post-training over skill-guided trajectories, which boosts Qwen3-VL-8B by 43% on VSI-Bench.