ASI-Bench:步骤比方法名更决定智能体表现

Rohan Paul · @rohanpaul_ai · X·2026-08-25 03:11·12小时前
AI 导读

ASI-Bench 用 60 个真实科研项目、11 个科学领域测试研究智能体,仅改变指令内容。18 种智能体与模型组合下,完整步骤提示平均得分 50.91,仅给方法名降至 29.10,再删方法名仅再降 2.5 分。仅方法名提示还多耗 59% token,因命名方法既限制方向又迫使智能体重建全部实现细节。

Rohan Paul@rohanpaul_ai
41AI 编辑部评分,满分 100

ASI-Bench:步骤比方法名更决定智能体表现

2026-08-25 03:11· 12小时前
AI 导读

ASI-Bench 用 60 个真实科研项目、11 个科学领域测试研究智能体,仅改变指令内容。18 种智能体与模型组合下,完整步骤提示平均得分 50.91,仅给方法名降至 29.10,再删方法名仅再降 2.5 分。仅方法名提示还多耗 59% token,因命名方法既限制方向又迫使智能体重建全部实现细节。

Telling a research AI agent which method to use is close to telling it nothing at all.

What actually carries performance is the procedure, so write the steps rather than the method name.

ASI-Bench gives agents 60 real research projects across 11 scientific fields. Same goal, same data, same scoring every time. Only the instructions change: full procedure, method name only, or nothing but the objective and the data.

Across 18 agent and model combinations, the average score fell from 50.91 with the full procedure to 29.10 with only the method named.

Dropping the method as well cost another 2.5 points. Nearly all the damage comes from losing the steps.

Method-only prompts were also the most expensive to run, burning 59% more tokens than complete instructions and more than prompts that named no method at all. Naming an approach pins the agent to a direction while still leaving it to rebuild every implementation detail.

来源:Rohan Paul· x.com