GUICrafter:利用海量未标注截图的弱监督GUI智能体
阅读原文· arxiv.orgGUICrafter是一个弱监督GUI智能体,通过两阶段课程学习框架降低对人工标注的依赖:阶段1利用大规模未标注截图和网页学习视觉定位,阶段2使用少量高质量数据通过强化学习校准。实验显示,GUICrafter仅使用UI-TARS 0.1%的数据即达到与之竞争甚至更优的性能;在相同标注数据量下,其表现超越所有先前方法(如GUI-R1)。代码、数据和模型已开源。
Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.