DailyReport:面向日常搜索任务的开放搜索智能体评估基准
阅读原文· arxiv.orgDailyReport 是一个用于评估搜索智能体(Search Agents)在日常搜索任务中能力的开放基准。它包含 150 个开放式任务和 3,546 条级联评分规则,将每个任务分解为子任务,并在可分离维度上进行细粒度评分。通过级联性能归因和以用户为中心的聚合,得到每个维度的可解释分数及用户偏好分数。在 17 个智能体系统上的测试结果显示,当前系统仍未达到用户期望。数据集和代码已公开。
Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses. For SAs evaluation, prior benchmarks mainly focus on specialized tasks that are unlikely to arise in real-world user scenarios. Moreover, their reliance on coarse task-level rubrics often limits evaluation interpretability. To bridge this gap, we introduce DailyReport, an open-ended benchmark to evaluate SA capabilities on daily search tasks. It contains 150 open-ended tasks with 3,546 associated rubrics, capturing widely discussed and timely information demands of real-world users. Each task is decomposed into subtasks and evaluated with cascade rubrics across disentangled dimensions. Through cascade performance attribution and user-centric aggregation, we derive highly interpretable scores for each dimension, along with a user preference score. Our results on 17 agentic systems show that current systems still fall short of users' expectations. To facilitate future research, our dataset and code are made publicly available at https://github.com/AGI-Eval-Official/DailyReport.