# AJ-Bench：面向环境感知评估的 Agent-as-a-Judge 基准测试

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-20 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo9u46qk04lhsls2n1re8g0l
- 原文链接：https://arxiv.org/abs/2604.18240

## AI 摘要

研究团队发布 AJ-Bench 基准测试，系统评估 Agent-as-a-Judge 在复杂环境中的验证能力。该基准涵盖搜索、数据系统和图形用户界面三大领域，包含155个任务与516条标注轨迹，全面测试评判智能体的信息获取、状态验证与过程验证能力。实验表明，Agent-as-a-Judge 相比 LLM-as-a-Judge 基线取得持续性能提升，但在基于智能体的验证中仍面临显著挑战。相关数据与代码已开源。

## 正文

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.