OpenAI 的 GPT-6 Astra 正在引发相互矛盾的基准测试结论。Epoch AI 认为它遥遥领先，而 Artificial Analysis 则评价它并不比前代更好。最大的惊喜来自 ARC-AGI-3，Astra 首次以高于人类平均水平的效率完成测试。ARC Prize 负责人 François Chollet 称这一进展比他预期的“快 2 倍”，并提前了自己的 AGI 预测时间表。
两家独立实验室各自将数十项单项测试合并为一个综合得分，却得出了相反的结论。Epoch AI 综合了 50 多项基准测试，将 GPT-6 Astra 以 169 分明显排在首位，领先于 267 个模型。Artificial Analysis 测试了知识、编程和文本理解能力，给 GPT-6 Astra 打出 61 分，与其前代完全持平，落后于 Claude Fable 5.1 的 66 分。
Astra 明显比其前代更贵。OpenAI 对每单位处理文本的收费是此前的 2.5 倍，这使得单个任务的成本比使用 Sol 时高出约 75%。与 Anthropic 相比，情况则相反。在编程任务上，根据 Artificial Analysis 的数据，Astra 取得了与 Claude Fable 5 相同的分数，但每个任务的成本不到后者的一半。原因在于该模型的节省程度。它只需要 Sol 所用计算步骤的三分之一，以及 Opus 5 所用计算步骤的五分之一。
| 基准测试 | Astra | Sol | Fable 5.1 | Opus 5 |
|---|---|---|---|---|
| Epoch ECI，综合得分 | 169 | 162 | 163 | 162 |
| AA 智能指数 | 61 | 61 | 66 | 63 |
| ARC-AGI-3，陌生游戏世界 | 62.7% | 7.8% | 无数据 | 30.2% |
| ARC-AGI-2，抽象视觉谜题 | 95.0% | 92.5% | 90.0% | 90.4% |
| ARC-AGI-1，旧版本 | 98.5%* | 97.5% | 97.5% | 97.5% |
| FrontierMath Erdős，开放数学 | 3% | 0% | 0% | 无数据 |
*GPT-6 Astra 在 xhigh 推理强度下达到此成绩；在最高强度下可达 97.5%。ARC-AGI-1 目前被认为已基本饱和。
在 Coding Agent Index 上，它以约为 Sol 三分之一的 token 用量取得 67 分，而 Fable 5.1 以 70 分领先。在 AA-Omniscience 上的幻觉率从 92% 降至 51%。与此同时，该模型在 GDPval-AA v2 上损失了约 80 个 Elo 积分，并在银行客服、SciCode 和长上下文推理任务上有所下滑。
Epoch AI 报告称，在新的 FrontierMath Erdős 基准上，GPT-6 Astra 是唯一在 68 个开放 Erdős 问题中解出两道、并通过 Lean 验证证明的模型，每次尝试的预算为 300 美元。另外还有三个解答来自非标准化的额外运行，消耗了超过 22 万美元的计算资源，但 Epoch 表示这些不计入得分。
细看 Epoch 的单项数据可以发现，GPT-6 Astra 与竞品 Fable 5.1 目前是在不同基准上被衡量的。Astra 在数学、知识和谜题方面领先。Fable 5.1 则在几乎所有编程测试中占据头名。但 Epoch 目前只记录了 Astra 的一项编程得分，而且该得分来自一次中等推理水平的运行。
ARC-AGI-3 出现大幅跃升
最明显的跃升出现在 ARC-AGI-3 上。该测试将 AI 投入陌生的游戏世界，没有人向它解释规则和目标。模型必须通过反复试错来弄清楚该做什么。GPT-6 Astra 以约 26,000 美元的测试成本达到了 62.7% 的得分。其前代 GPT-5.6 Sol 仅取得 7.78%，竞品 Claude Opus 5 则拿到 30.16%。Fable 5 和 Fable 5.1 尚未登上该基准。
OpenAI 报告的 99.9% 是在不同条件下取得的。在该设置中，Astra 得以使用 OpenAI 自建的 harness，该 harness 会在单个请求之间保留推理链，并自动对长时间运行进行总结。根据 ARC Prize 的测量，与在内部 harness 上的运行相比，这些运行速度约为 3.66 倍，token 消耗减少 49%，该对比基于两种设置均成功解决的 167 组游戏推理对。
这些 harness 的使用以及随之而来的性能跃升，在 GPT-5.6 Sol 时期就已经成为 ARC Prize 与 OpenAI 之间的一个争议点。ARC Prize 指出，只有在内部 ARC harness 上运行的较低数值 62.7% 才允许各厂商之间进行公平比较，不过它也计划未来公布厂商 harness 上的数据。
在这里，更多思考反而降低了账单
思考强度与成本之间的关系并不寻常。通常情况下，推理级别越高，测试运行的成本就越高。而Astra的情况恰恰相反。在标准ARC脚手架（scaffold）上，成本从无推理时的49,791美元下降到最高推理级别时的26,098美元，同时得分从35.2%攀升至62.7%。据ARC Prize称，原因是Astra能用更少的步数解决游戏，这意味着更少的模型调用和更少的token消耗。有一个异常现象格外突出：“低”级别的得分为17.5%，比完全无推理运行时的表现还要差。ARC Prize未对这一离群值发表评论，但GPT-6 Astra在其他基准测试中表明，由于其新架构，它可以在不进行推理的情况下解决更长周期（longer-horizon）的任务——该架构推测会在生成第一个token之前在内部进行循环处理。
作为对比，人类测试者每场90分钟的测试获得115美元报酬，外加每解决一个游戏5美元，因此按大约9次尝试计算，每个游戏的成本约为12.78美元。但这主要支付的是时间和参与意愿。如果只把大脑消耗的代谢能量折算成电能，ARC Prize得出的结果是每个游戏0.067美分。
比原始得分更有趣的是效率。在发布之前，ARC Prize让约500名测试者在不进行预筛选的情况下进行游戏，并记录了每个级别中成功解决者的步数中位数。在使用OpenAI脚手架（scaffold）的运行中，Astra以低于该中位数的步数通过了96%的关卡，平均步数略高于中位数的一半。与通常的成本衡量指标不同，这一数字并不追踪消耗的计算量。它追踪的是模型在掌握某个环境之前，需要多少与该环境相关的经验。
这正是组织者原本预期人类能长期保持优势的地方。对于暴力求解方法来说依然如此，但在顶级模型身上，ARC Prize 看到的几乎是一种二元模式。一旦模型弄清了其中的机制，它的执行水平就会落入人类的高效区间。
Astra 发明了自己的记号系统
为了达到这一水平，Astra 会保留自己的笔记，并自行发明一套类似代数的速记法，用来记录对象、坐标、规则和未完成的计划，例如把 `extend8 to3; retract10 to2` 记作一系列有序的移动，或把 `Turn 5: P=(24,20), empty, facing west` 记作一条状态笔记。ARC Prize 在其他模型身上也观察到了类似行为，但特别指出 Astra 在精确性和信息密度方面尤为突出。在标准测试框架下，这是一项重要技能，因为模型没有保存到自己可见笔记中的一切信息都会丢失。
ARC 联合创始人 François Chollet 在 X 上将其描述为“针对每个游戏和关卡进行的高效、即时符号化世界建模”。该模型甚至做到了“开发出自己的速记 DSL 来表示游戏内情境”，其核心“本质上是一种针对特定游戏的代数记号系统”。对 Chollet 来说，最重要的在于这种行为从何而来：“Astra 展现出了我们此前只在精密测试框架中见过的符号化建模行为，因此测试框架的能力正越来越多地转移到模型本身之中。”
Astra 创建了一个密集、紧凑的符号化世界模型，以完成 ARC-AGI-3 环境。
例如，在 s5i5 环境中，Astra：
- 记录了当前关卡、中心朝向和机关长度：“L8：中心 q2（8↓）。长度：14=1……”
- 它将操作映射到精确的控制上：… pic.twitter.com/tMHP002mkB
— ARC Prize（@arcprize）2026年9月3日
第三个测试环境展示了 Astra 在使用外部工具时的能力。PRO-LONG 是由第三方开发的智能体框架，ARC Prize 团队在早期将其部署为 ARC-AGI-3 的红队测试伙伴——也就是说，用它来系统性地探索该基准测试的极限。与标准设置不同，在该环境中，模型获得了一个沙盒，可以在其中执行自己的代码。
Astra 充分利用了这一点，为每个游戏编写了小型程序库：游戏棋盘解析器、状态模型、搜索算法和规划器。在一个有守卫的迷宫游戏中，它先后开发了寻路器、战斗规则模块、巡逻移动模型，以及一个不断将自己的预测与观察结果进行对比的脚本。ARC Prize 未观察到任何试图逃逸沙盒的行为。这些运行与人类测试条件不具备可比性，因为人类受试者既没有代码解释器，也没有记事本。这里衡量的是模型与自建工具的综合表现。
Chollet：没有 AGI 的证据，但速度超出预期
ARC Prize 明确表示，不会将这些结果解读为通用人工智能（AGI）的证据。“到目前为止，我们对这个系统的全部了解就是它的基准测试分数，”Chollet 写道。在 ARC-AGI-3 发布时，他们在每一场演示中都强调了一点：“解决它并不能证明 AGI。它并非旨在成为一条终点线。”虽然该基准测试确实检验了 AGI 系统应具备的正确质性特征——即不确定性下的探索、无指导下的适应能力，以及从稀疏数据中进行因果世界建模——但它是在“小规模”上进行的。这些游戏运行的时间尺度比现实世界任务短数个数量级，因此所需的数据更少、建模复杂度更低，所需的即时学习也更少。
大约六个月前，ARC-AGI-3 发布时，Chollet 在回应一个关于“饱和”的问题时曾表示“大约一年”，具体取决于攻克该基准测试的方法有多聚焦。因此，Astra 的到来“比预期快了大约两倍”。“我相信，进步的速度将令许多人惊讶，新一代模型的能力将挑战人们基于前几代模型所形成的对 AI 的认知。”当有用户问他早前对 2030 年实现 AGI 的预测是否仍然成立时，Chollet 简洁地回答道：“会更早，因为进展速度比我预期的要快。”
结果又是另一个基准测试。据 Chollet 称，ARC-AGI-4 自 ARC-AGI-3 发布以来一直在开发中，计划于 2027 年第一季度发布。基准测试是一个持续的过程，它随模型的演进而演进，始终瞄准 AI 与人类智能之间尚存的差距。该组织认为 ARC-AGI-3 本身相当有限：确定性机制、封闭式目标，且无法表征开放的真实世界。下一代基准测试除其他方向外，旨在探索递归式自我改进和开放式创新。
继续阅读以了解全貌。订阅以获取无炒作报道。
- 完整访问 THE DECODER 上的每一篇文章
- 无广告
- 参与评论与社区讨论
- 每周通过邮件获取 AI 新闻摘要
- 每年 6 期：“AI Radar”——深度剖析最重要的 AI 议题
- 每日 AI 新闻，始终紧跟最新动态
- 我们完整的十年存档
- 由拥有 10 年以上 AI 经验的团队撰写报道
OpenAI's GPT-6 Astra is drawing contradictory benchmark verdicts. Epoch AI puts it out in front, while Artificial Analysis rates it no better than its predecessor. The biggest surprise comes from ARC-AGI-3, where Astra works more efficiently than the average human for the first time. ARC Prize chief François Chollet calls the progress "2x faster" than he expected and is moving up his AGI forecast.
Two independent labs each roll dozens of individual tests into a single overall score, but they reach opposite conclusions. Epoch AI combines more than 50 benchmarks and puts GPT-6 Astra clearly in first place with 169 points, ahead of 267 models. Artificial Analysis tests knowledge, coding, and text comprehension, and rates GPT-6 Astra at 61 points, exactly level with its predecessor and behind Claude Fable 5.1 at 66 points.
Astra is clearly more expensive than its own predecessor. OpenAI charges two and a half times as much per unit of processed text, which makes a task cost roughly 75 percent more than it did with Sol. Compared with Anthropic, the picture flips. On coding tasks, Astra hits the same score as Claude Fable 5 according to Artificial Analysis, but costs less than half as much per task. The reason is how sparing the model is. It needs only a third of the compute steps Sol uses and a fifth of what Opus 5 uses.
| Benchmark | Astra | Sol | Fable 5.1 | Opus 5 |
|---|---|---|---|---|
| Epoch ECI, overall score | 169 | 162 | 163 | 162 |
| AA Intelligence Index | 61 | 61 | 66 | 63 |
| ARC-AGI-3, unfamiliar game worlds | 62.7% | 7.8% | no data | 30.2% |
| ARC-AGI-2, abstract visual puzzles | 95.0% | 92.5% | 90.0% | 90.4% |
| ARC-AGI-1, older version | 98.5%* | 97.5% | 97.5% | 97.5% |
| FrontierMath Erdős, open math | 3% | 0% | 0% | no data |
*GPT-6 Astra at xhigh reasoning effort; at max it hits 97.5 percent. ARC-AGI-1 is now considered largely saturated.
On the Coding Agent Index, it reaches 67 points at roughly a third of Sol's token usage, while Fable 5.1 leads with 70. The hallucination rate on AA-Omniscience drops from 92 to 51 percent. At the same time, the model loses about 80 Elo points on GDPval-AA v2 and slips on banking support, SciCode, and long-context reasoning tasks.
Epoch AI reports that on the new FrontierMath Erdős, GPT-6 Astra was the only model to solve two of 68 open Erdős problems with Lean-verified proofs, on a budget of $300 per attempt. Three more solutions came out of non-standardized extra runs that burned through more than $220,000 in compute, but Epoch says those don't count toward the score.
A closer look at Epoch's individual numbers shows that GPT-6 Astra and rival Fable 5.1 have so far been measured on different ground. Astra leads on math, knowledge, and puzzles. Fable 5.1 holds the top marks on nearly every coding test. But Epoch has recorded only a single coding score for Astra so far, and that one comes from a run at a medium reasoning level.
ARC-AGI-3 shows a big jump
The clearest jump comes on ARC-AGI-3. The test drops an AI into unfamiliar game worlds whose rules and goals nobody explains to it. The model has to figure out what to do by trial and error. GPT-6 Astra reaches 62.7 percent at a test cost of roughly $26,000. Its predecessor GPT-5.6 Sol managed 7.78 percent, and rival Claude Opus 5 got 30.16 percent. Fable 5 and Fable 5.1 aren't on the benchmark yet.
The 99.9 percent OpenAI reported came under different conditions. In that setup Astra got to use the harness OpenAI built, which keeps reasoning chains between individual requests and automatically summarizes long runs. By ARC Prize's measurements, those runs went about 3.66 times faster and used 49 percent fewer tokens than runs on the in-house harness, compared across 167 game-reasoning pairs that both setups solved.
The use of these harnesses, and the performance jump that comes with them, was already a sticking point between ARC Prize and OpenAI with GPT-5.6 Sol. ARC Prize notes that only the lower figure of 62.7 percent, run on the internal ARC harness, allowed a fair comparison between vendors, though it plans to publish the numbers from vendor harnesses in the future as well.
Here, more thinking lowers the bill
The relationship between thinking effort and cost is unusual. Normally a higher reasoning level makes a test run more expensive. With Astra it's the opposite. On the standard ARC scaffold, costs drop from $49,791 with no reasoning to $26,098 at maximum reasoning, while the score climbs from 35.2 to 62.7 percent. According to ARC Prize, the reason is that Astra solves the games in fewer moves, which means fewer model calls and fewer tokens. One oddity stands out: the "low" level scores 17.5 percent, worse than running with no reasoning at all. ARC Prize doesn't comment on the outlier, but GPT-6 Astra in other benchmarks showed that it can solve longer-horizon tasks without reasoning due to its new architecture which presumably loops processing internally before generating the first token.
For comparison, the human testers got $115 per 90-minute session plus $5 per game solved, so at about nine attempts that works out to roughly $12.78 per game. But that mostly pays for time and willingness to take part. Count only the metabolic energy of the brain as electricity instead, and ARC Prize arrives at 0.067 cents per game.
More interesting than the raw score is the efficiency. Before the launch, ARC Prize had about 500 testers play with no pre-screening and recorded, for each level, the median number of moves among those who solved it. On the run with the OpenAI scaffold, Astra cleared 96 percent of levels in fewer moves than that median, on average with a little over half. Unlike the usual cost measures, this figure doesn't track compute consumed. It tracks how much experience with an environment the model needed before it mastered it.
This is exactly where the organizers had expected humans to hold a lasting edge. That still holds for brute-force approaches, but with top models ARC Prize sees an almost binary pattern. Once the model has figured out the mechanics, its execution lands in the human efficiency range.
Astra invents its own notation
To get there, Astra keeps its own notes and works out a self-invented, algebra-like shorthand in which it records objects, coordinates, rules, and open plans, for example extend8 to3; retract10 to2 as an ordered sequence of moves or Turn 5: P=(24,20), empty, facing west as a state note. ARC Prize saw similar behavior from other models, but singles out Astra for its precision and information density. On the standard harness, that's an important skill, because everything the model doesn't save into its own visible notes is lost.
ARC co-founder François Chollet describes it on X as "highly efficient, on-the-fly symbolic world modeling for each game and level." The model goes so far as "developing its own shorthand DSL to represent in-game situations," which at its core is "essentially a game-specific algebraic notation." What matters most to Chollet is where this behavior comes from: "Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses, so harness capabilities are increasingly shifting into the model itself."
Astra creates a dense, compact symbolic world model to complete ARC-AGI-3 environments.
For example, in environment s5i5, Astra:
- Recorded the current level, hub orientation, and mechanism lengths: “L8: hub q2 (8↓). Lengths: 14=1…”
- It mapped operations to exact controls:… pic.twitter.com/tMHP002mkB
— ARC Prize (@arcprize) September 3, 2026
A third test environment demonstrates what Astra is capable of with external tools. PRO-LONG is an agent framework developed by a third party that the ARC Prize team deployed early on as a red-teaming partner for ARC-AGI-3—that is, to systematically explore the limits of the benchmark. Unlike in the standard setup, the model is provided with a sandbox in which it can execute its own code.
Astra took advantage of this and wrote small program libraries for each game: parsers for the game board, state models, search algorithms, and planners. In a maze game featuring guards, a pathfinder, a combat rules module, a model of patrol movements, and a script that continuously compared its own predictions against observations were developed one after another. ARC Prize did not observe any attempts to escape from the sandbox. These runs are not comparable to the human test conditions, since the test subjects had neither a code interpreter nor a notepad. What is being measured here is the combined performance of the model and the self-built tools.
Chollet: No Proof of AGI, but Faster Than Expected
ARC Prize explicitly does not interpret the results as evidence of general artificial intelligence. “All we know about the system so far are its benchmark scores,” writes Chollet. When ARC-AGI-3 was launched, they emphasized one point in every presentation: “Solving it is not proof of AGI. It is not intended as a finish line.” While the benchmark does test the correct qualitative properties expected of an AGI system—namely, exploration under uncertainty, adaptation without guidance, and causal world modeling from sparse data—it does so “on a small scale.” The games ran on time scales that were orders of magnitude shorter than real-world tasks and consequently required less data, less modeling complexity, and less on-the-fly learning.
When ARC-AGI-3 was released about six months ago, Chollet had responded to a question about saturation by saying “about a year,” depending on how focused the approach to the benchmark was. Astra, therefore, arrived “about twice as fast” as expected. “I believe the pace of progress will surprise many people, and what the new models are capable of will challenge the perception of AI that people have formed based on earlier generations of models.” When asked by a user whether his earlier AGI forecast for 2030 still held, Chollet replied succinctly: “Sooner, because progress is happening faster than I expected.”
The result is another benchmark. According to Chollet, ARC-AGI-4 has been in development since the release of ARC-AGI-3 and is scheduled for release in the first quarter of 2027. Benchmarking is an ongoing process that evolves alongside the models and always targets the remaining gap between AI and human intelligence. The organization considers ARC-AGI-3 itself to be quite limited: deterministic mechanics, closed-ended goals, and no representation of the open real world. The next generation is intended to explore, among other things, recursive self-improvement and open innovation.
Read on for the full picture.
Subscribe for hype-free coverage.
- Full access to every article on THE DECODER
- No ads
- Join the comments and community discussions
- A weekly AI news recap via mail
- 6x/year: "AI Radar" — deep dives on the AI topics that matter most
- Daily AI news, always up to date
- Our full ten-year archive
- Covered by a team with 10+ years in AI