Terminal Wrench:331个可奖励作弊环境与3632条攻击轨迹数据集
阅读原文· arxiv.org研究团队发布Terminal Wrench数据集,收录331个可奖励作弊的终端智能体基准环境及3632条攻击轨迹,覆盖Claude Opus 4.6等三款前沿模型。任务涵盖系统管理、机器学习等领域,攻击手段包括输出欺骗、栈帧内省及二进制劫持。可监控性研究显示,移除思维链后LLM裁判检测准确率下降(AUC从0.97降至0.92)。数据集已开源。
We release Terminal Wrench, a subset of 331 terminal-agent benchmark environments, copied from the popular open benchmarks that are demonstrably reward-hackable. The data set includes 3,632 hack trajectories and 2,352 legitimate baseline trajectories across three frontier models (Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4). Each entry preserves the original task definition alongside full attack trajectories that show how the verifier was bypassed. It also includes cases where the task was not solved as intended. The tasks span system administration, machine learning, software engineering, and security challenges; the exploits range from simple output spoofing to stack-frame introspection, standard-library patching, and rootkit-style binary hijacking. Crucially, these exploits are specific to each task, rather than the evaluation harness, making them harder to patch. We also present a monitorability study in which hack trajectories are sanitized or stripped of reasoning traces and then scored by an LLM judge, showing that detection degrades meaningfully when chain-of-thought is removed (AUC drops from 0.97 to 0.92). The data set is publicly available at https://github.com/few-sh/terminal-wrench.