# QuoteBench：匹配分数如何掩盖命令路径失败

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-08-13 08:00
- AIHOT 分数：52
- AIHOT 链接：https://aihot.virxact.com/items/cmt3566580hhvro6tgqk0w8yv
- 原文链接：https://arxiv.org/abs/2608.13547

## AI 摘要

QuoteBench 用 56 个单次任务区分命令生成错误与执行传输失败。新增解析器使成功率下降 55.4 至 73.2 个百分点，披露边界后六个配置恢复 30.4 至 60.7 点。GPT-5.6-sol 的匹配差距 -3.6 点掩盖了 -64.3 点损伤与 +60.7 点补偿，部署配置在 26 对可比模型中产生一次明确排序反转。

## 正文

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.