InSight: 通过可控VLA实现自主技能获取
阅读原文· arxiv.orgInSight是一个框架,通过让视觉-语言-动作(VLA)模型在原始动作层面(如“将夹爪移动到碗边”“向上抬起”)变得可控,实现自主技能获取。包含两个阶段:(1)自动化分割管道,利用VLM规划分解和末端执行器位姿将演示分割为带标签原始动作;(2)VLM引导的数据飞轮,识别缺失原始动作,自主尝试并标注存储成功演示。在模拟和真实操作任务(方块翻转、抽屉关闭、清扫、扭转、倾倒)上的评估表明,无需人类演示目标技能即可习得,且原始动作可组合执行新任务。
Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills in the training data. We present InSight, a framework that unlocks autonomous skill acquisition by rendering VLAs steerable at the primitive-action level (e.g., "move gripper to the bowl", "lift upward", "pour the bottle"). InSight consists of two primary stages: (1) an automated segmentation pipeline that partitions demonstrations into labeled primitives via VLM plan decomposition and end-effector poses to enable VLA primitive steerability, and (2) a VLM-guided data flywheel that identifies missing primitives required to accomplish a novel task, autonomously attempts demonstrations of the missing primitives with VLM-proposed low-level control, and automatically labels, stores, and integrates successful demonstrations into the VLA training set. We evaluate InSight across simulation and real-world manipulation tasks, including block flipping, drawer closing, sweeping, twisting, and pouring, without any human demonstrations of these target skills. Once learned, these primitives can be composed to execute novel, long-horizon tasks without additional human demonstrations. Our findings demonstrate that primitive steerability provides a practical foundation for continual skill acquisition in VLA policies. Project website: https://insight-vla.github.io.