elvis@omarsar0
37AI 编辑部评分,满分 100

Harness-IF:量化编码智能体规则真实影响

2026-08-14 02:00· 28分钟前
AI 导读

Harness-IF 通过逐条评分 256 条规则并从执行证据中剔除巧合,量化编码智能体对 AGENTS.md 规则的遵循度。在 12 个前沿模型上,原始准确率 72.1-85.9%,剔除先验后的准确率降至 66.1-78.6%,每模型下降 3.6-7.4 点。研究发现规则优先级不随提示深度变化,系统提示、项目文件和用户指令均高于工具与技能描述。

If you write rules in an AGENTS.md, this one is worth your time.

When a coding agent follows your rule, it may have been going to do that anyway.

Harness-IF separates the two by scoring 256 rules one at a time from execution evidence, then re-running every task with the rule withheld across nine probe builds to find which rules actually oppose the model's defaults.

Across 12 frontier models, raw accuracy runs 72.1 to 85.9%, and Against-Prior Accuracy runs 66.1 to 78.6%. Every model gets worse once coincidence is stripped out, by 3.6 to 7.4 points.

One finding worth flagging. Precedence does not follow prompt depth. System prompts, project files, and user instructions all outrank tool and skill descriptions.

Paper: https://arxiv.org/abs/2608.11727

Track more trending AI papers in our academy: https://academy.dair.ai/

来源:elvis · x.com

Harness-IF:量化编码智能体规则真实影响

elvis · @omarsar0 · X·2026-08-14 02:00·28分钟前
AI 导读

Harness-IF 通过逐条评分 256 条规则并从执行证据中剔除巧合,量化编码智能体对 AGENTS.md 规则的遵循度。在 12 个前沿模型上,原始准确率 72.1-85.9%,剔除先验后的准确率降至 66.1-78.6%,每模型下降 3.6-7.4 点。研究发现规则优先级不随提示深度变化,系统提示、项目文件和用户指令均高于工具与技能描述。

If you write rules in an AGENTS.md, this one is worth your time.

When a coding agent follows your rule, it may have been going to do that anyway.

Harness-IF separates the two by scoring 256 rules one at a time from execution evidence, then re-running every task with the rule withheld across nine probe builds to find which rules actually oppose the model's defaults.

Across 12 frontier models, raw accuracy runs 72.1 to 85.9%, and Against-Prior Accuracy runs 66.1 to 78.6%. Every model gets worse once coincidence is stripped out, by 3.6 to 7.4 points.

One finding worth flagging. Precedence does not follow prompt depth. System prompts, project files, and user instructions all outrank tool and skill descriptions.

Paper: https://arxiv.org/abs/2608.11727

Track more trending AI papers in our academy: https://academy.dair.ai/

来源:elvis· x.com