# Benchmark Agent：全自主评测基准构建系统

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-04 08:00
- AIHOT 分数：64
- AIHOT 链接：https://aihot.virxact.com/items/cmq13ksuq0c50sltrlxhs2qqa
- 原文链接：https://arxiv.org/abs/2606.06462

## AI 摘要

Benchmark Agent 是一个全自主智能体系统，可端到端完成评测基准构建，涵盖用户查询分析、子任务设计、数据标注与质量控制。系统一次性生成 15 个代表性基准，覆盖文本理解、多模态理解和领域特定推理等评估场景。人工评估、LLM-as-a-judge 和一致性检验表明，Benchmark Agent 能以极少量人工参与产出高质量评测样本。持续评估还发现当前模型在某些领域推理任务上仍有明显短板。预览页面与代码即将公开。

## 正文

Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.