GUI-CIDER: 通过因果内化与密度感知样本重选进行中期训练的GUI智能体
阅读原文· arxiv.org针对多模态大语言模型构建的GUI智能体因缺乏操作世界知识导致任务完成能力受限的问题,本文提出GUI-CIDER中期训练方法。该方法通过三个阶段显式内化知识:首先从GUI轨迹中合成静态规划与动态因果知识文本;然后通过奖励因果结构、惩罚语义冗余进行样本重选;最后使用精炼数据进行中期训练以嵌入知识。实验表明,该方法一致提升了智能体对GUI操作的理解与任务成功率。代码已开源。
Despite the rapid progress of multimodal large language models in building Graphical User Interface (GUI) agents, their real-world task completion is fundamentally bottlenecked by a lack of world knowledge about GUI operations. Existing solutions typically rely on expensive multi-agent scaffolding or conventional post-training paradigms, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). However, post-training only allows agents to implicitly absorb world knowledge through action annotations or reward signals, leading to inefficient trajectory memorization rather than genuine comprehension. Therefore, an approach that enables explicit learning of this knowledge is imperative. To this end, we propose GUI-CIDER, a mid-training method that explicitly internalizes GUI world knowledge through Causal Internalization and Density-aware Exemplar Reselection. GUI-CIDER operates in three stages: (1) data synthesis, which distills static planning and dynamic causal knowledge from GUI trajectories into text; (2) exemplar reselection, which filters the corpus by rewarding causal structures and penalizing semantic redundancy; and (3) mid-training, where the refined data is used to embed the acquired knowledge. Extensive experiments on two GUI knowledge benchmarks and three task completion benchmarks demonstrate that GUI-CIDER consistently improves both the agent's understanding of GUI operations and its task success rates.The codes are available at https://github.com/Wuzheng02/GUI-CIDER.