Artificial Analysis@ArtificialAnlys

2026-06-16 09:51·16天前

AI 摘要

Artificial Analysis 发布 Intelligence Index v4.1，转向智能体任务。升级 Terminal-Bench 2.1、τ³-Bench Banking、GDPval-AA v2（Elo 重基线、引入前沿模型评审、回合上限增至250），移除饱和的 IFBench。新增每任务成本、时间、输出 token 指标及缓存 token 影响。关键结果：Claude Fable 5（60分）领先但不可用；可用模型中 Claude Opus 4.8（max）56分居首，GPT-5.5（xhigh）55分。开源 DeepSeek V4 Pro 与 MiniMax M3 均44分。成本方面，Opus 4.8 每任务 $1.78，GPT-5.5 $0.99，DeepSeek V4 Pro 仅 $0.04。时间方面，Grok 4.3 最快（1.5分钟），Opus 4.8 需6.4分钟，GPT-5.5 需3.7分钟，Gemini 3.1 Pro Preview 以1.6分钟得46分。

Announcing Artificial Analysis Intelligence Index v4.1： a shift toward agentic workloads， featuring upgraded benchmarks and new per-task metrics

The Artificial Analysis Intelligence Index is our synthesis metric for assessing model intelligence and tracking AI progress. v4.1 marks a broader shift toward agentic workloads， with three main changes：

Updated and reweighted evaluations toward agentic tasks： 1. We upgraded three evaluations， removed one， and reweighted the Intelligence Index： ➤ Upgraded Terminal-Bench Hard to Terminal-Bench 2.1 and τ2-Bench Telecom to τ3-Bench Banking. Both move to newer， more robust task sets with harder， more realistic agentic scenarios that better separate frontier models ➤ Upgraded GDPval-AA to GDPval-AA v2. The upgrade re-baselines Elo to human performance at 1000， introduces a rotating panel of frontier-model judges， and raises the turn limit from 100 to 250 for longer-horizon agent trajectories ➤ Removed IFBench due to saturation. The benchmark no longer distinguishes frontier models sufficiently， so we have removed it from the Intelligence Index. We will continue to run it and publish results on new model releases

2. Cost per Task， Time per Task， and Tokens per Task： Three new per-task metrics， reported for every model and based on the Intelligence Index. We take the total cost， total time， and total output tokens for a model to run the Intelligence Index and divide by the number of tasks across its evaluations， giving the average cost， time， and output tokens to complete a single Intelligence Index task

3. Cached input token reporting： We now report cached input tokens and their impact on cost， including the cost to run the Intelligence Index， to better reflect the real cost of running each model

Artificial Analysis@ArtificialAnlys · X

60导出 Markdown

2026-06-16 09:51·16天前

在 X 看原推· x.com

AI 摘要

Announcing Artificial Analysis Intelligence Index v4.1： a shift toward agentic workloads， featuring upgraded benchmarks and new per-task metrics