ReVision:通过时序视觉冗余缩减扩展计算机使用智能体规模
阅读原文· arxiv.org计算机使用智能体(CUA)依赖GUI截图,每张截图编码大量视觉token,长轨迹下token成本激增。ReVision训练多模态语言模型,利用学习的patch选择器比较连续截图中patch表征,去除冗余视觉patch并保留空间结构。基于Qwen2.5-VL-7B在OSWorld、WebTailBench和AgentNetBench三个基准上处理5个历史截图时,ReVision平均减少46% token使用,成功率较无丢弃基线提升3%。该效率使智能体能用更少token处理更长轨迹,且随着移除冗余后纳入更多历史观测,性能持续改善。
Computer-use agents (CUAs) rely on visual observations of graphical user interfaces, where each screenshot is encoded into a large number of visual tokens. As interaction trajectories grow, the token cost increases rapidly, limiting the amount of history that can be incorporated under fixed context and compute budgets. This has resulted in no or very limited improvement in the performance when using history unlike other domains. We address this inefficiency by introducing ReVision, which is used to train multimodal language models on trajectories where redundant visual patches are removed using a learned patch selector that compares patch representations across consecutive screenshots while preserving spatial structure required by the model. Across three benchmarks, OSWorld, WebTailBench, and AgentNetBench, when processing trajectories with 5 history screenshots using Qwen2.5-VL-7B, ReVision reduces token usage by 46% on average while improving success rate by 3% over the no drop baseline. This establishes a clear efficiency gain, enabling agents to process longer trajectories with fewer tokens. With this improved efficiency, we revisit the role of history in CUAs and find that performance continues to improve as more past observations are incorporated when redundancy is removed.