正文 · AI 翻译
Artificial Analysis 与 IBM 软件创新实验室联合推出 ITBench-AA，这是评估模型在智能体企业 IT 任务上表现的全新基准系列中的首个基准，首先从站点可靠性工程任务开始，前沿模型在该任务上的得分低于 50%。ITBench-AA 的 SRE 任务对模型在 Kubernetes 事件响应方面的性能进行基准测试，模型和智能体必须通过读取日志、追踪依赖关系以及识别复杂基础设施中的根因实体来诊断实时系统。底层 ITBench 数据集由 IBM 开发，充分利用了其在企业 IT 运营方面的深厚专业知识。Artificial Analysis 在过去 6 个月中与 IBM 密切合作，为该数据集开发了适用于前沿 AI 评估的实现方案，从站点可靠性工程（SRE）开始，并逐步扩展到财务运营（FinOps）和首席信息安全官（CISO）任务。
关键发现：
- Claude Opus 4.7（自适应推理，最大努力）以 47% 的得分领先，其次是 GPT-5.5（xhigh）的 46% 和 Qwen3.7 Max 的 42%。
- 所有前沿模型的得分均低于 50%，这使得 ITBench-AA SRE 成为我们基准套件中饱和度最低的智能体基准之一。作为参考，前沿模型在 Terminal-Bench 上的得分要高得多。
- 交互轮次数量差异接近 3 倍，且更长的推理轨迹并不会转化为更高的准确率。GPT-5.5（xhigh）每项任务平均 31 轮，准确率为 46%，而 Gemini 3.1 Pro Preview 平均 83 轮，准确率仅为 30%。过度调查的模型往往会将上游故障注入机制或并发症状误报为假阳性。
- GLM-5.1（推理）以 40% 的得分领先于开放权重模型，与 Gemini 3.5 Flash（高）基本持平。DeepSeek V4 Pro（推理，最大努力）以 38% 紧随其后，Gemma 4 31B（推理）为 37%，领先于 Gemini 3.1 Pro Preview 的 30%。
ITBench-AA SRE 概览：
- 总共 59 个 SRE 任务：40 个公开任务和 19 个全新的、未公开的任务
- 每个任务提供一个 Kubernetes 事件快照，其中包含告警、事件、追踪、指标、日志和应用拓扑。模型必须识别出导致该事件的、最小的一组独立根因 Kubernetes 实体。
- 故障涵盖典型的 SRE 故障模式，包括基础设施、服务、应用以及混沌注入事件，例如资源配额耗尽、发布失败、连接池耗尽和网络分区。方法论细节如下：
- 智能体框架：每个任务由运行在我们开源 Stirrup 参考框架中的模型解决，该模型具有对包含相关日志和快照的沙盒文件系统的 shell 访问权限。每个任务上限 100 轮交互，每个任务重复 3 次。
- 模型和智能体提交一份它们认为导致事件的根因实体列表（Kubernetes 部署、服务、Pod 等）。每次提交都会与 IBM 提供的真实根因集合进行比对。
- 评分采用全召回率下的平均精确率：如果模型遗漏了任何真实根因，则该次重复得分为 0.0。如果模型识别出所有真实根因，则其得分等于其精确率——即提交的实体中实际为根因的比例，也就是真正例 /（真正例 + 假正例）。最终得分为 59 个任务 × 3 次重复的平均值。
- 该框架（Stirrup）在所有评估模型中保持不变，从而允许模型之间进行公平对比。
亮点
- 任务要求智能体通过 shell 命令调查 Kubernetes 事件快照，并提交结构化的 JSON 诊断结果，以识别负责的根因实体。在一个公开的 SRE 任务中，智能体在前端路径中观察到面向用户的故障。它使用 shell 命令检查离线快照：查看告警以确定事件窗口，然后通过追踪/日志将故障范围缩小到前端流量。拓扑图定位受影响的服务，Kubernetes 清单显示存在一个阻止前端的网络策略。成功的诊断识别出负责的根因实体：otel-demo/NetworkPolicy/frontend-block-all-ports。
- 更多的交互轮次并不等同于更好的答案。那些提交了超出真正根本原因之外额外相关实体的模型会受到惩罚：在召回率约束下的精确率指标中，识别出正确根本原因但额外添加了上游机制（例如 chaos-mesh 控制器）或伴随症状，会被视为误报。这就是为什么一些长轨迹模型的得分反而低于更简洁的模型：Gemini 3.1 Pro Preview 平均 83 轮，得分 30%；而 Gemma 4 31B（推理版）平均 58 轮，得分 37%。
- 开放权重模型处于 ITBench-AA SRE 评测的成本前沿。Gemma 4 31B（推理版）每个任务成本 0.14 美元，得分 37%，在得分和成本两方面均优于 Gemini 3.1 Pro Preview（每个任务 2.23 美元，得分 30%）。GLM-5.1（推理版）每个任务成本 1.23 美元，得分 40%，以更低成本达到了与 Gemini 3.5 Flash（高）相同的得分（后者成本 1.70 美元）。Claude Opus 4.7（自适应推理，最大努力模式）以 47% 的得分领跑排行榜，但也是成本最高的，每个任务需 5.38 美元。
ITBench-AA 是与 @IBM 合作，基于其 ITBench 基准构建的。
- 更多信息请参见：ITBench 论文（arXiv）：https://arxiv.org/abs/2502.05352
- GitHub：https://github.com/itbench-hub/ITBench
- ITBench-AA 排行榜：https://artificialanalysis.ai/evaluations/itbench-aa
- ITBench-AA HuggingFace 仓库：https://huggingface.co/datasets/ArtificialAnalysis/ITBench-AA/tree/main/sre
Artificial Analysis and IBM Software Innovation Lab are launching ITBench-AA, the first in a new series of benchmarks evaluating models on agentic enterprise IT tasks, starting with Site Reliability Engineering tasks where frontier models score below 50% ITBench-AA’s SRE tasks benchmark model performance on Kubernetes incident response, where models and agents must diagnose live systems by reading logs, tracing dependencies, and identifying root-cause entities across complex infrastructure. The underlying ITBench dataset has been developed by IBM, leveraging deep expertise in enterprise IT operations. Artificial Analysis has worked closely with IBM over the last 6 months to develop an implementation of the dataset for frontier AI evaluation, beginning with Site Reliability Engineering (SRE) and expanding to Financial Operations (FinOps) and Chief Information Security Officer (CISO) tasks over time. 
Key findings:
- Claude Opus 4.7 (Adaptive Reasoning, Max Effort) leads at 47%, followed by GPT-5.5 (xhigh) at 46% and Qwen3.7 Max at 42%.
- All frontier models score below 50%, making ITBench-AA SRE one of the least saturated agentic benchmarks in our suite. For context, frontier models score considerably higher on Terminal-Bench.
- Turn counts vary nearly 3x and longer trajectories do not translate to higher accuracy. GPT-5.5 (xhigh) averages 31 turns per task at 46%, while Gemini 3.1 Pro Preview averages 83 turns at 30%. Models that over-investigate tend to surface upstream fault-injection mechanisms or co-occurring symptoms as false positives.
- GLM-5.1 (Reasoning) leads open weights models at 40%, effectively tied with Gemini 3.5 Flash (high). DeepSeek V4 Pro (Reasoning, Max Effort) follows at 38%, with Gemma 4 31B (Reasoning) at 37%, ahead of Gemini 3.1 Pro Preview at 30%.
ITBench-AA SRE overview:
- 59 SRE tasks in total: 40 public tasks and 19 brand new, held-out tasks
- Each task provides a Kubernetes incident snapshot containing alerts, events, traces, metrics, logs, and application topology. The model must identify the minimal set of independent root-cause Kubernetes entities responsible for the incident.
- Faults span typical SRE failure modes including infrastructure, service, application, and chaos-injected incidents, such as resource quota exhaustion, rollout failures, connection pool exhaustion, and network partitions. Methodology details:
- Agentic harness: each task is solved by the model running in our open-source Stirrup reference harness, with shell access to a sandboxed file system containing the relevant logs and snapshots. 100-turn cap per task, 3 repeats per task.
- Models and agents submit a list of root-cause entities (Kubernetes Deployments, Services, Pods, etc.) they believe caused the incident. Each submission is compared against a ground-truth set of root causes provided by IBM.
- Scoring uses average precision at full recall: if a model misses any of the ground-truth root causes, it scores 0.0 for that repeat. If it identifies all of them, it is awarded a score equal to its precision - the share of its submitted entities that are actual root causes, i.e. true positives / (true positives + false positives). The headline score is the average across 59 tasks × 3 repeats.
- The harness (Stirrup) is held constant across all evaluated models, allowing an apples-to-apples comparison between models.
Highlights
- Tasks require agents to investigate Kubernetes incident snapshots through shell commands and submit a structured JSON diagnosis identifying the responsible root-cause entities. In one public SRE task, the agent sees user-facing failures in the frontend path. It uses shell commands to inspect the offline snapshot: reviewing alerts shows the incident window, then traces/logs narrow the failure to frontend traffic. Topology pins down the affected services, and Kubernetes manifests reveal a network policy blocking the frontend. The successful diagnosis identifies the responsible root-cause entity: otel-demo/NetworkPolicy/frontend-block-all-ports.
- More turns do not mean better answers. Models that submit additional contributing entities beyond the true root cause get penalized: identifying the correct root cause but adding upstream mechanisms (e.g., a chaos-mesh controller) or co-occurring symptoms counts as a false positive under recall-gated precision. This is why some models with long trajectories underperform terser ones: Gemini 3.1 Pro Preview averages 83 turns and scores 30%, while Gemma 4 31B (Reasoning) averages 58 turns and scores 37%.
- Open weights models sit on the cost frontier of ITBench-AA SRE. Gemma 4 31B (Reasoning) scores 37% at $0.14 per task, outperforming Gemini 3.1 Pro Preview ($2.23 per task, 30%) on both score and cost. GLM-5.1 (Reasoning) scores 40% at $1.23 per task, matching Gemini 3.5 Flash (high) ($1.70) on score at lower cost. Claude Opus 4.7 (Adaptive Reasoning, Max Effort) leads the leaderboard at 47% but is the most expensive at $5.38 per task.
ITBench-AA is built in partnership with @IBM based on their ITBench benchmark.
- For more information see: ITBench paper on arXiv: https://arxiv.org/abs/2502.05352
- GitHub: https://github.com/itbench-hub/ITBench
- ITBench-AA leaderboard: https://artificialanalysis.ai/evaluations/itbench-aa
- ITBench-AA HuggingFace repo: https://huggingface.co/datasets/ArtificialAnalysis/ITBench-AA/tree/main/sre