网络是复杂、开放且不断变化的，这使得为视觉网络智能体扩展训练数据变得极具挑战性。现有的数据收集尝试仍局限于用于监督微调的离线轨迹，或用于强化学习训练的少量模拟环境，因此未能捕捉到网络的多样性。我们提出了 Weblica（Web Replica），一个用于构建可复现且可扩展网络环境的框架。我们的框架利用 1) HTTP 级别的缓存来捕获和重放稳定的视觉状态，同时保留交互行为；2) 基于大语言模型的环境合成，该合成以真实世界网站和核心网络导航技能为基础。利用此框架，我们将强化学习训练扩展到数千个多样化的环境和任务。我们最好的模型 Weblica-8B 在多个网络导航基准测试中，以更少的推理步骤超越了相似规模的开源权重基线模型，在测试时计算量增加时表现出良好的扩展性，并且与 API 模型具有竞争力。
相关阅读与更新。
重述网络：一种实现计算与数据高效语言建模的方法
大语言模型是在大规模网络抓取数据上训练的，这些数据通常是非结构化、嘈杂且措辞不佳的。当前的缩放定律表明，从这类数据中学习需要大量的计算资源和数据，且需求会随着所训练模型的规模增长而增加。这之所以不可行，既是因为预训练带来的巨大计算成本和时长，也是因为网络上高质量数据即将面临稀缺。在这项研究中……
重述网络：一种实现计算与数据高效语言建模的方法
本文已被 ICLR 2024 的基础模型数据问题研讨会（Data Problems for Foundation Models workshop）接收。
大语言模型是在大规模网络抓取数据上训练的，这些数据通常是非结构化、嘈杂且措辞不佳的。当前的缩放定律表明，从这类数据中学习需要大量的计算资源和数据，且需求会随着所训练模型的规模增长而增加。这之所以不可行，既是因为巨大的计算成本和时长……
The web is complex, open-ended, and constantly changing, making it challenging to scale training data for visual web agents. Existing data collection attempts remain limited to offline trajectories for supervised fine-tuning or a handful of simulated environments for RL training, thus failing to capture web diversity. We propose Weblica (Web Replica), a framework for constructing reproducible and scalable web environments. Our framework leverages 1) HTTP-level caching to capture and replay stable visual states while preserving interactive behavior and 2) LLM-based environment synthesis grounded in real-world websites and core web navigation skills. Using this framework, we scale RL training to thousands of diverse environments and tasks. Our best model, Weblica-8B, outperforms open-weight baselines of similar size across multiple web navigation benchmarks while using fewer inference steps, scales favorably with additional test-time compute, and is competitive with API models.
Related readings and updates.
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this…
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
This paper has been accepted at the Data Problems for Foundation Models workshop at ICLR 2024.
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration…