AGORA:一个基于档案的智能体工作场所文档推理基准
阅读原文· arxiv.org大语言模型正越来越多地作为智能体在文档上进行推理,而非依赖参数化知识。Agora基准测试包含362个问题、8个领域共9664份真实文档和3.72亿个模型token,远超任何模型的上下文窗口,迫使智能体进行审慎探索而非穷举扫描。评估8个模型后,最强模型准确率仅为59.4%,且各领域表现差异显著。
Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window, so agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.