暗处进化的智能体：基于自我偏好的回顾式能力包优化
潘文博
刘树杰
林钦佑
曾静莹
唐先锋
周向阳
陆艳
贾小华
香港城市大学
微软亚洲研究院
摘要
AI 智能体依赖由技能、工具和工作流组成的能力包来解决复杂问题。持续改进这一能力包对于适应新任务至关重要。然而，现有的优化方法通常需要真实标注的验证集，但在实际部署场景中，这类标注数据很难获取。为解决这一问题，我们提出了回顾式能力包优化（RHO），这是一种仅利用历史轨迹来优化智能体能力包的自监督方法。具体来说，RHO 从历史轨迹中选取一个多样化的困难任务核心集，并并行地重新求解这些任务。智能体通过自我验证和自我一致性来分析这些 rollout 结果，然后生成候选的能力包更新方案，并通过自身的成对自我偏好选择出最有效的方案。我们在软件工程、技术工作和知识工作这三个不同领域对 RHO 进行了评估。值得注意的是，仅经过一轮优化，SWE-Bench Pro 上的通过率就从 59% 提升到了 78%，且无需任何外部评分。此外，我们的分析表明，RHO 能有效针对先前的失败模式。因此，优化后的能力包改变了智能体的行为模式，并在长周期会话中保持了更高的准确率。代码已开源至 https://github.com/wbopan/retro-harness，项目网站为 https://paper-rho.wenbo.io。
暗处进化的智能体：基于自我偏好的回顾式能力包优化
潘文博¹ 刘树杰² 林钦佑² 曾静莹² 唐先锋² 周向阳² 陆艳² 贾小华¹ ¹香港城市大学 ²微软亚洲研究院
1 引言
一个控制框架（harness）通过为AI智能体提供可用的技能、工作流程和工具，使其能够完成复杂任务。一个重要的研究课题是如何持续改进这个控制框架。具体来说，在智能体部署之后，我们的目标是让其控制框架通过从过往经验中学习而不断进化，从而提升其在未来任务上的表现。
先前的研究已经提出了多种进化智能体控制框架的方法（Zhou 等人，2022；Yang 等人，2023；Khattab 等人，2023；Yuksekgonul 等人，2024；Agrawal 等人，2025；Hu 等人，2024；Lee 等人，2026）。然而，这些方法都依赖于针对验证集进行评分来指导改进。在实际部署场景中，往往很难收集到一个能准确估计未来任务分布以验证更新后控制框架的验证集。另一方面，智能体的持续运行会自然地从过往任务中产生丰富的轨迹数据。这引出了我们的核心问题：当我们只能获取过往轨迹时，能否改进智能体控制框架以提升未来性能？
为了解决这个问题，我们提出了回溯式控制框架优化（RHO），这是一种通过回溯分析过往轨迹来优化控制框架的自监督方法。该方法利用智能体内部对轨迹的自我偏好来指导优化过程。图1将RHO与传统的验证反馈优化（即针对带标签的验证集进行迭代优化）进行了对比。
图 2 展示了这一流程。具体而言，给定大量历史轨迹，RHO 首先选取一个多样且具有挑战性的核心任务集。然后，智能体对核心集中的每个任务进行多次重试，以生成并行轨迹。在此基础上，我们提取两种诊断信号，即轨迹内的自我验证和跨并行轨迹的自我一致性。这些信号随后被用于指导工具集更新的生成。最后，通过利用智能体的成对自我偏好，我们从新生成的提案中选出最有前景的工具集。
我们在涵盖软件工程、技术工作和知识工作的三个智能体领域评估了 RHO 的有效性。RHO 在所有三个领域均持续提升了智能体的性能。值得注意的是，通过对软件工程轨迹进行单轮回顾式工具集优化，我们将 SWE-Bench Pro（Deng 等人，2025）上的通过率从 59% 提升至 78%，且无需依赖验证集进行评分。
此外，我们详细分析了回顾式优化流程如何提升性能。我们观察到，RHO 针对过去任务中遇到的典型失败模式，设计了特定的技能和工具。这些组件重塑了智能体的行动模式，帮助其在长周期会话中保持更高的准确性。此外，我们定量分析了回顾过程中诊断信号的贡献。该分析表明，RHO 中的每一步都逐步分离出有助于性能提升的信号。
-
我们提出了回顾式工具集优化，该方法弥补了仅从无标签轨迹中改进完整工具集（包括记忆、上下文、技能和工具）的空白。
-
我们在三种场景下评估了 RHO，结果表明，在可比预算下，回顾式分析始终优于直接的经验积累，并超越了基于验证反馈驱动的进化方法。
-
我们提供了关于工具链优化对智能体性能影响的定量分析，表明收集有效的改进信号能够引导对工具链进行针对性调整，并优化智能体的行为。
2 相关工作
工具链优化。
工具链优化通过编辑固定模型周围的提示词、程序参数或工作流代码来改进智能体。一条研究路线针对标注指标优化提示词或流水线参数，涵盖大语言模型作为优化器的搜索（Yang 等人，2023）、声明式流水线编译（Khattab 等人，2023）、文本梯度更新（Yuksekgonul 等人，2024）以及反思式提示词演化（Agrawal 等人，2025）。另一条更具智能体特性的路线则让元智能体重写智能体自身的代码，其中 ADAS 搜索智能体系统设计空间（Hu 等人，2024），而 Meta-Harness 则利用先前候选方案的执行轨迹和得分来搜索工具链代码（Lee 等人，2026）。尽管这些方法在编辑的表层上有所不同，但它们都通过标注的验证指标来引导搜索。RHO 则偏离了这一范式，它不需要验证反馈，仅通过对未标注的过往轨迹进行一次回顾性处理即可改进工具链。
智能体自我改进。
第二条路线利用智能体对轨迹的自我判断替代真实标签，使智能体能够从自身过往经验中实现改进。Dynamic Cheatsheet 在测试时维护一个自我管理的记忆库，其中包含可复用的策略和代码片段（Suzgun 等人，2025）；而 ReasoningBank 则从自我判断的成功与失败案例中提炼出可泛化的推理策略（Ouyang 等人，2025）。MemMA 协调多个智能体的记忆周期，并针对自身生成的探测问题修复其记忆库（Lin 等人，2026）；Sleep-time Compute 在查询到达之前离线预计算有用的上下文（Lin 等人，2025）；M⋆ 则将记忆系统本身作为一个可执行程序进行演化，为每个任务族发现特定于任务的记忆框架（Pan 等人，2026a）。与我们工作同期进行的 SkillOS 则利用结果奖励和评判奖励通过强化学习训练一个技能策展器，从累积的经验中更新技能库（Ouyang 等人，2026）。这些方法丰富了智能体的存储记忆、上下文或技能列表，但未触及框架的其他部分。相比之下，RHO 对完整的框架（包括可执行工具和指令）进行优化，而不仅仅是记忆。附录 A 提供了与相关工作的详细对比。
3 问题设定
我们将工具包（harness）定义为智能体可用于解决任务的工具、提示词和技能的持久化集合。给定一个任务和一个工具包，智能体可以通过推理、行动和观察的循环来尝试完成任务。这个多步骤过程会生成一条轨迹（trajectory），记录智能体读取的信息、其思维链、使用的工具以及最终输出。我们将这个执行过程称为带提示词的智能体操作。当智能体执行多个任务时，它会生成一个轨迹数据集。这些轨迹通常包含失败实例和可用于改进工具包的有用见解。因此，我们提出疑问：智能体能否回顾性地分析过去的轨迹，以优化其工具包并提升未来表现？为量化这一点，我们定义了一个潜在效用函数，用于衡量轨迹的质量。我们将优化过程形式化为一个返回修改后工具包的函数。目标是找到最优工具包，以最大化未来任务的期望效用：
问题。然而，在实践中准确估计这个效用函数是困难的。要评估工具包的真实效用，我们需要一个具有代表性的未来任务验证集，以及一种计算使用该特定工具包的智能体成功率的方法。在我们的设定中，该函数是潜在的，无法直接观测。
我们的方法。由于效用是潜在的，我们无法直接对其进行优化。相反，我们用一种自我偏好估计器来替代这个潜在效用。具体来说，我们指示智能体比较同一任务上的多条轨迹，以计算自我偏好排序。我们定义了一个排序函数。该函数对给定轨迹产生偏好顺序，并提供解释，说明智能体为何偏好某些执行过程而非其他。下一节将详细介绍我们如何组织求解、排序和优化操作，以提升潜在工具包效用。
4 回顾式 Harness 优化
我们提出 RHO，这是一种自监督方法，仅利用历史轨迹即可改进 harness。具体来说，我们的流程（图 2）包含三个阶段，即核心集选择、分组展开和最佳 harness 提案。首先，我们选择一个具有代表性的历史任务子集来定义优化目标。接着，针对该核心集中的每个任务，我们采样一组并行展开，并从中提取 harness 改进信号。最后，我们基于这些信号生成候选 harness，并通过成对自偏好保留最受青睐的一个。完整算法详见算法 1。
4.1 核心集选择
给定大量历史轨迹，我们需要提取最关键信号来引导线束优化。对每一条单独轨迹进行线束优化在计算上不可行，而且还会用琐碎信号稀释重要信号。为解决这一问题，我们首先从完整集合中选取一个核心子集，用以代表最需要优化的轨迹。具体而言，我们要求该核心子集既能捕捉高难度场景，也能覆盖多样化场景。这一要求促使我们的优化在解决最困难问题时能涵盖广泛的失败模式。为此，我们引入行列式点过程（DPP）核（Kulesza 和 Taskar，2012），在满足多样性约束的同时，按难度对所有历史轨迹进行排序。实际操作中，我们使用语言模型裁判分析每条轨迹，提取难度分数并附带文本描述。该描述详细说明问题的具体挑战和潜在失败模式。随后我们计算该描述的嵌入向量，并以两个轨迹嵌入向量之间的余弦相似度作为相似性度量。综合考虑难度分数和轨迹相似性矩阵后，我们构建一个核矩阵。
其中 是轨迹难度分数 的缩放版本：
借助该核函数 ，DPP 以与核行列式 成正比的概率选取一个子集，并通过参数 调节难度与多样性的相对重要性（ 越大越侧重难度）。当 时，轨迹完全按难度排序；当 （均匀权重）时，完全按相似性多样性排序。利用 ，我们将轨迹选入一个核心子集，使其覆盖困难且多样化的失败模式，供后续阶段使用。
4.2 分组展开
受先前工作中利用组内相对优势作为强化学习奖励信号（Shao 等人，2024）的启发，我们在每个核心集任务上通过并行运行智能体求解来生成一组轨迹。随后，智能体比较这些组内轨迹，以识别表现不佳的运行。然后，智能体利用组内的对比信号来制定优化工具链的指令。具体而言，我们沿着两个维度执行这种自我偏好分析。
-
自我验证。该维度检查每个轨迹内智能体的正确性。智能体根据所需任务和环境观察结果检查每个轨迹，以确定目标是否被高效达成，这利用了模型识别自身知识局限的部分能力（Pan 等人，2025）。在此过程中，它会标记出错误的工具调用、错误的假设以及过早停止。这些被标记的方面随后被提取出来，作为相对表现不佳的运行需要改进的地方。
-
自我一致性。该维度检查智能体的行为在不同轨迹之间是否保持一致。由于低自我一致性通常表明高不确定性（Wang 等人，2022；Farquhar 等人，2024），我们指示智能体分析轨迹之间的矛盾。智能体识别出关键性的分歧，例如不同的计划、工具序列或最终答案，并生成优化指令以鼓励更一致的行为。
这些分析会生成 JSON 格式的结构化评估，对于每个任务，它们的并集构成了改进指令。因此，我们合并核心集中所有任务的指令，形成最终的工具链改进指令。
| 工具链 | SWE-Bench Pro | Terminal-Bench 2 | GAIA-2 | ||||
| 方法 | 架构 | 通过率 | 通过率 | 通过率 | |||
| Vanilla Codex | 无 | 0.59 | 不适用 | 0.71 | 不适用 | 0.29 | 不适用 |
| Dynamic Cheatsheet (Suzgun 等人，2025) | 技能 | 0.62 | 0.73 | 0.30 | |||
| ReasoningBank (Ouyang 等人，2025) | 记忆 | 0.61 | 0.73 | 0.28 | |||
| 睡眠时间计算（Lin 等人，2025） | 记忆 | 0.64 | 0.73 | 0.32 | |||
| RHO | 技能+工具 | 0.78 | 0.76 | 0.37 | |||
4.3 最佳候选框架方案
在获取改进指令后，我们通过将这些指令提供给智能体来优化框架。然而，正如先前关于智能体进化的研究所观察到的（Agrawal 等人，2025；Hu 等人，2024；Lee 等人，2026），框架优化本质上是随机的，即使输入信号有效，也可能无法可靠地提升性能。为缓解这一局限，我们并行采样多个框架方案，并使用智能体自我偏好对其进行筛选。这种筛选旨在偏向那些改进能够泛化到未来任务的候选方案。具体而言，我们执行并行优化调用以生成候选框架，记为 至 。随后，我们使用这些候选方案在核心集任务上获取新的轨迹集合。对于每个核心集任务，我们通过将每个候选框架的新轨迹与原始框架的旧轨迹进行排序，来计算一个智能体偏好分数。我们跨核心集汇总这些分数，以确定每个候选方案的相对优势分数：
其中 是任务 的原始框架轨迹。最后，我们返回具有最大相对优势的候选框架来替换原始框架。只有当该分数严格大于零（）时，我们才接受此更新。
5 实验与结果
实验设置。我们使用 Codex 智能体（OpenAI，2025）作为回溯优化的基础框架。具体来说，该智能体配置了高推理强度的 GPT-5.5（OpenAI，2026）。当调用 Codex 解决任务时，我们将框架构建为一个可配置的工作区文件夹。该文件夹包含作为工具的可执行脚本，以及用于技能和指令的文本文件。在所有实验中，我们将核心集大小设置为 10。此外，并行轨迹采样和框架提案均使用 3。为衡量改进效果，我们分别使用原始 Codex 框架和优化后的框架，报告在保留测试集上的通过率。
数据。我们从现有基准数据集中收集历史轨迹。具体来说，我们将原始基准划分为轨迹集和测试集。然后，我们在轨迹集上运行原始 Codex 智能体，为 RHO 生成所需的轨迹。在此基础上，我们在 SWE-Bench Pro、Terminal-Bench 2 和 GAIA-2 上评估 RHO。SWE-Bench Pro 包含需要仓库级推理和多文件编辑的长周期软件工程任务（Deng 等人，2025）。Terminal-Bench 2 包含带有可执行评分器的命令行任务（Terminal-Bench 团队，2025）。GAIA-2 在动态、异步环境中评估 LLM 智能体的知识工作能力（Froger 等人，2026）。因此，这三个基准涵盖了软件工程、技术工作和知识工作等多种任务类型。我们在附录 E 中提供了关于基准和数据划分的详细信息。
5.1 与无反馈基线的比较
我们将 RHO 与三种不需要验证反馈的竞争性工具优化方法进行了比较。作为基线方法，Dynamic Cheatsheet 会持续记录有用的事实和流程（Suzgun 等人，2025）。ReasoningBank 存储可复用的推理模式，并在推理时检索最相关的条目（Ouyang 等人，2025）。类似地，Sleep-time Compute 将过去的轨迹离线预处理为紧凑的笔记，然后将其添加到智能体的上下文前面（Lin 等人，2025）。我们将每种基线方法适配到我们的数据集和智能体设置中，同时保持总智能体调用预算大致固定，以确保公平比较。详细的适配过程见附录 F。
如表 1 所示，RHO 在所有三个基准测试中都带来了持续的改进，而基线方法则没有。最值得注意的是，我们在 SWE-Bench Pro 上实现了 19% 的绝对提升，且不依赖任何基于验证的评分。我们将这一优势归因于 RHO 实现的更灵活的工具优化。具体来说，智能体可以为工具创建新的工具、技能和指令，而之前的方法主要侧重于记忆系统或基于文本的技能。此外，自我偏好的使用可能有助于这些工具改进的一致性。相比之下，基线方法的性能提升往往较小，且在不同数据集上有所差异。在下一节（第 5.2 节）中，我们将研究 RHO 如何修改工具以改进智能体。
5.2 优化后的工具包含什么
图 3 总结并解释了 RHO 优化后生成的新工具内容。在我们的工作中，工具被实现为一个目录，其中包含用于指令和技能的 Markdown 文件，以及用于工具的可执行脚本。
在所有三个基准测试中，RHO 都为测试框架增加了多项新技能和新工具。其中许多新增内容针对的是原始框架常见的失败模式。例如，在 SWE-Bench Pro 中，智能体了解到 Go 工具链位于默认路径之外的一个非标准位置。它还发现，在生成最终差异补丁之前，必须清理 Python 缓存目录，因为不这样做往往会导致补丁无法干净地应用。为了解决这些问题，智能体新增了一个 `check_build_and_lint` 工具，该工具可以定位非标准工具链，并标记出必须排除在补丁之外的生成产物，从而修复了原始轨迹中反复遗漏的差异补丁卫生规范。这些例子说明了 RHO 如何通过分析过去的失败，在不同场景中识别出有用的工具和技能。
5.3 与基于验证反馈的优化方法的比较
接下来，我们将 RHO 与 Meta-Harness（Lee 等人，2026）进行比较。Meta-Harness 是一种基于验证反馈的优化器，它会提出测试框架的修改方案，在标注的验证集上对每个候选方案进行评分，并保留验证通过率最高的修改方案。为了保持公平比较，我们使用与 Meta-Harness 相同的 Codex 智能体作为提议者和求解器。与 RHO 不同，Meta-Harness 需要保留的标注数据，并且由于它需要多轮迭代，因此消耗的智能体调用次数更多。因此，我们在单轮迭代下对其进行评估以匹配我们的计算预算，同时也在扩展预算设置下（运行十轮）对其进行评估。
| 方法 | 验证标签 | 智能体调用次数 | SWE-Bench Pro |
| RHO | 无 | 103 () | 0.78 |
| Meta-Harness（1 轮） | 需要 | 41 () | 0.62 |
| Meta-Harness（10 轮） | 需要 | 320 () | 0.80 |
在匹配的单轮预算下，Meta-Harness 使用验证分数选择其最佳候选方案，但在 SWE-Bench Pro 上仅达到 0.62 的通过率。这远低于 RHO 所实现的 0.78 通过率。将 Meta-Harness 扩展到 10 轮设置后，其在 SWE-Bench Pro 上的性能上限提升至 0.80。然而，这种更高的性能需要大约三倍于 RHO 的优化阶段计算量，更重要的是，它仍然依赖于 RHO 所不使用的保留标签。
6 讨论
6.1 优化后智能体行为如何变化？
尽管 RHO 为智能体创造了新的技能和工具，但这些更新究竟通过何种机制使智能体在未来任务中表现更佳，并非一目了然。为了探究这一点，我们可视化了工具调用的频率以及相对于智能体所执行步骤数的累积成功率。具体来说，图 4 绘制了在给定智能体步骤数内解决的保留任务的累积比例，并展示了智能体的动作组合如何随时间变化。我们观察到，所有三个数据集的性能提升主要源于在需要长程规划的任务上取得了更高的成功率。相比之下，性能增益集中在长程任务上，而非那些在较少步骤内完成的任务，这一点在 SWE-Bench Pro 上最为明显。此外，优化过程改变了智能体的工作模式。因此，优化后的智能体更倾向于更严重地依赖特定类型的动作。例如，在 SWE-Bench Pro 上，智能体验证其工作的频率大大提高。这种主动验证似乎占据了长程任务性能提升的很大一部分。在 Terminal-Bench 2 和 GAIA-2 上，智能体通过主动应用新开发的工具来提高其准确性。作为我们动作层面统计数据的补充，对长推理链进行 token 层面的归因分析，为研究长程智能体行为提供了一个更细粒度的视角（Pan 等人，2026b）。
6.2 核心集选择如何塑造能力演化？
数据集 均值 选定值 标准差 最低值 SWE-Bench Pro 0.79 0.78 0.06 0.73 Terminal-Bench 2 0.74 0.76 0.03 0.71 GAIA-2 0.34 0.37 0.03 0.32
变体 SWE Pro TB 2 GAIA-2 完整诊断 0.78 0.76 0.37 自一致性 0.56 0.75 0.27 自验证 0.70 0.73 0.30 原始轨迹 0.60 0.75 0.29
我们研究了核心集选择如何影响优化过程。为此，我们将基于DPP的选择方法与几种消融策略进行了比较。这些变体包括：仅根据难度选择任务、纯粹以最大化覆盖率为目标选择任务，以及随机采样轨迹。此外，我们还测量了每种选择策略下优化后工具链的最终性能。如图5中任务嵌入向量的t-SNE投影所示，仅根据难度选择任务会导致所选样本聚集在任务分布的一个狭窄区域内。这种聚集现象的发生，是因为语言模型判定某些类型的任务本质上更难，从而未能将其他任务类型纳入核心集。因此，该策略在优化后未能带来有意义的性能提升。同样，仅针对覆盖率进行优化也会产生次优结果。相比之下，随机采样有时能够选出对优化有用的轨迹。这些发现表明，一种能够平衡难度与多样性的核心集选择策略，对于为引导工具链优化提供恰当的信号至关重要。
6.3 RHO能否产生一致的工具链更新？
由于 RHO 在没有真实标签作为参考的情况下运行，我们评估其优化输出在不同运行之间是否一致。这种一致性与我们最佳候选方案提案的整体有效性密切相关。我们考察选择策略是否能够可靠地识别出在下游任务中表现最佳的候选方案。在本实验中，我们测量所有三个生成的候选方案的测试分数，而不仅仅是得分最高的那个。表 3 显示，生成的候选方案仅表现出中等程度的方差。值得注意的是，即使是得分最低的候选方案，也能在基线基础上显著提升智能体性能。此外，最佳候选选择机制防止了表现不佳的方案被部署。具体来说，在所有三个基准测试中，被选中的方案得分均高于最差的候选方案。同时，得分最高的候选方案并不总是与测试集上得分最高的候选方案一致，不过该选择机制始终能够避开最差的候选方案。
6.4 回顾性分析贡献有多大？
我们分析了回顾性分析过程中提取的两种信号（即自我验证和自我一致性）的贡献。我们将这些显式信号与一种更直接的方法进行了比较，后者在优化过程中提供原始轨迹，并跳过显式的回顾性分析步骤。此外，我们还研究了每种诊断信号对最终性能的单独贡献。
为了探究分组展开和诊断阶段的必要性，我们进行了一项消融研究。具体来说，我们分别移除了自我验证和自我一致性信号，随后重新运行了优化和评估流程。我们还引入了一个原始轨迹基线。该基线跳过了独立的排名分析，而是直接将原始轨迹提供给优化步骤，要求智能体在单次处理中分析轨迹并提出改进方案。表4显示，无论是移除自我一致性信号还是自我验证信号，都会持续降低各基准测试的最终性能。这一结果表明，这两个信号对于优化“约束框架”都至关重要。此外，在所有三个基准测试上，完整诊断的表现均优于简化的原始轨迹基线。这表明，尽管单次轨迹分析是可行的，但明确的自我验证和自我一致性信号并非可有可无，而是必不可少的，它们能在不同数据集上带来更可靠的改进。
7 结论
我们引入了RHO，它将约束框架改进重新定义为一个回顾性过程——智能体可以基于自身过往经验来执行，而非依赖外部真实反馈进行搜索。其核心思想是：智能体自身的轨迹已经包含了改进所需的信号，因为重新解决过往任务并比较结果，能够揭示约束框架在何处失效以及如何修复。在所有三个领域（软件工程、技术工作和知识工作）中，这种自监督循环都能带来持续的、在未见数据上的性能提升，并重塑智能体的工作方式。我们将RHO视为朝着智能体能够从部署中积累的经验中持续改进（而标注验证数据稀缺）这一目标迈出的一步。
局限性
在本文中，我们介绍了 RHO，一种无需任何外部评分、仅利用智能体自身过往轨迹来提升其工具包的自监督方法。然而，在没有真实反馈的情况下运行存在若干局限性。首先，群体回放会多次重放每个核心集任务，这假设环境能够干净地重置并容忍重复尝试，从而将一次性或不可逆的任务排除在 RHO 所针对的场景之外。其次，RHO 假设智能体能力的相当一部分是由可编辑的提示词、技能和工具包所介导的；我们的实验涵盖了软件工程、技术工作和知识工作，将 RHO 扩展到具有不同工具包界面、任务特征和回放预算的领域仍是未来的工作。第三，RHO 仅信任过往轨迹作为其优化输入；在开放环境中，这些轨迹可能嵌入任务中途注入的对抗性内容（Liu 等人，2026），而从被篡改的轨迹中提炼出的工具包更新可能会固化此类行为，这强化了伦理声明中审计实践的必要性。
伦理声明
RHO 根据模型生成的判断来修改智能体的持久行为。如果评估者偏好错误的偏好、不安全的程序或带有偏见的规则，这可能会放大这些问题。部署时应保留完整的审计日志，对敏感的工具包编辑需经人工批准，并在将已接受的工具包应用于高影响任务之前，使用特定领域的安全检查。
可复现性声明
每次运行都会持久化保存提示词、补全结果、轨迹、诊断信息、候选工具包、工具包差异、配置、评分、运行元数据以及保留的测试报告。本文中的数字直接来自记录的运行报告。与相关工作的详细比较见附录 A，提示词列表见附录 B，超参数见附录 C，完整流水线细节见附录 D，各数据集规范见附录 E，基线实现见附录 F，各方法优化阶段的计算成本见附录 G。
参考文献
- Agrawal 等人（2025 年）Lakshya A. Agrawal、Shangyin Tan、Dilara Soylu、Noah Ziems、Rishi Khare、Krista Opsahl-Ong、Arnav Singhvi、Herumb Shandilya、Michael J. Ryan、Meng Jiang、Christopher Potts、Koushik Sen、Alexandros G. Dimakis、Ion Stoica、Dan Klein、Matei Zaharia 和 Omar Khattab。2025 年。《GEPA：反思性提示词进化可超越强化学习》。arXiv 预印本 arXiv:2507.19457。
- Deng 等人（2025 年）Xiang Deng、Jeff Da、Edwin Pan、Yannis Yiming He、Charles Ide、Kanak Garg、Niklas Lauffer、Andrew Park、Nitin Pasari、Chetan Rane、Karmini Sampath、Maya Krishnan、Srivatsa Kundurthy、Sean Hendryx、Zifan Wang、Vijay Bharadwaj、Jeff Holm、Raja Aluri、Chen Bo Calvin Zhang 及其他 3 位作者。2025 年。《SWE-Bench Pro：AI 智能体能否解决长周期软件工程任务？》arXiv 预印本 arXiv:2509.16941。
- Farquhar 等人（2024 年）Sebastian Farquhar、Jannik Kossen、Lorenz Kuhn 和 Yarin Gal。2024 年。《利用语义熵检测大语言模型中的模型幻觉》。Nature，630：625–630。
- Froger 等人（2026 年）Romain Froger 及其他 1 位作者。2026 年。《Gaia2：在动态和异步环境中基准测试大语言模型智能体》。收录于国际学习表征会议（ICLR）。ArXiv:2602.11964。
- Hu 等人（2024 年）Shengran Hu、Cong Lu 和 Jeff Clune。2024 年。《智能体系统的自动化设计》。arXiv 预印本 arXiv:2408.08435。
- Khattab 等人（2023 年）Omar Khattab、Arnav Singhvi、Paridhi Maheshwari、Zhiyuan Zhang、Keshav Santhanam、Sri Vardhamanan、Saiful Haq、Ashutosh Sharma、Thomas T. Joshi、Hanna Moazam、Heather Miller、Matei Zaharia 和 Christopher Potts。2023 年。《DSPy：将声明式语言模型调用编译为自我改进的流水线》。arXiv 预印本 arXiv:2310.03714。
- Kulesza 和 Taskar（2012 年）Alex Kulesza 和 Ben Taskar。2012 年。《用于机器学习的行列点过程》。机器学习基础与趋势，5(2–3)：123–286。
- Lee 等人（2026 年）Yoonho Lee、Roshen Nair、Qizheng Zhang、Kangwook Lee、Omar Khattab 和 Chelsea Finn。2026 年。《Meta-Harness：模型 harness 的端到端优化》。arXiv 预印本 arXiv:2603.28052。
- Lin 等人 (2025) Kevin Lin、Charlie Snell、Yu Wang、Charles Packer、Sarah Wooders、Ion Stoica 和 Joseph E. Gonzalez。2025 年。《睡眠时间计算：超越测试时的推理扩展》。arXiv 预印本 arXiv:2504.13171。
- Lin 等人 (2026) Minhua Lin、Zhiwei Zhang、Hanqing Lu、Hui Liu、Xianfeng Tang、Qi He、Xiang Zhang 和 Suhang Wang。2026 年。《MemMA：通过多智能体推理和原位自我进化协调记忆周期》。arXiv 预印本 arXiv:2603.18718。
- Liu 等人 (2026) Zhichao Liu、Wenbo Pan、Haining Yu、Ge Gao、Tianqing Zhu 和 Xiaohua Jia。2026 年。《WebTrap：在导航过程中对浏览器智能体进行隐蔽的中途劫持》。arXiv 预印本 arXiv:2605.08310。
- OpenAI (2025) OpenAI。2025 年。《OpenAI Codex》。https://developers.openai.com/codex/。
- OpenAI (2026) OpenAI。2026 年。《推出 GPT-5.5》。https://openai.com/index/introducing-gpt-5-5/。
- Ouyang 等人 (2026) Siru Ouyang、Jun Yan、Yanfei Chen、Rujun Han、Zifeng Wang、Bhavana Dalvi Mishra、Rui Meng、Chun-Liang Li、Yizhu Jiao、Kaiwen Zha、Maohao Shen、Vishy Tirumalashetty、George Lee、Jiawei Han、Tomas Pfister 和 Chen-Yu Lee。2026 年。《SkillOS：为自我进化智能体学习技能策展》。arXiv 预印本 arXiv:2605.06614。
- Ouyang 等人 (2025) Siru Ouyang、Jun Yan、I-Hung Hsu、Yanfei Chen、Ke Jiang、Zifeng Wang、Rujun Han、Long T. Le、Samira Daruki、Xiangru Tang、Vishy Tirumalashetty、George Lee、Mahsan Rofouei、Hangfei Lin、Jiawei Han、Chen-Yu Lee 和 Tomas Pfister。2025 年。《ReasoningBank：用推理记忆扩展智能体自我进化》。arXiv 预印本 arXiv:2509.25140。
- Packer 等人 (2023) Charles Packer、Sarah Wooders、Kevin Lin、Vivian Fang、Shishir G. Patil、Ion Stoica 和 Joseph E. Gonzalez。2023 年。《MemGPT：迈向将大语言模型作为操作系统》。arXiv 预印本 arXiv:2310.08560。
- Pan 等人 (2026a) Wenbo Pan、Shujie Liu、Xiangyang Zhou、Shiwei Zhang、Wanlu Shi、Mirror Xu 和 Xiaohua Jia。2026a 年。《M⋆：每个任务都应有自己的记忆框架》。arXiv 预印本 arXiv:2604.11811。
- Pan 等人 (2026b) Wenbo Pan、Zhichao Liu、Xianlong Wang、Haining Yu 和 Xiaohua Jia。2026b 年。《迈向长程可解释性：推理型大语言模型的高效且忠实的多 token 归因》。arXiv 预印本 arXiv:2602.01914。
- Pan 等人 (2025) Wenbo Pan, Jie Xu, Qiguang Chen, Junhao Dong, Libo Qin, Xinfeng Li, Haining Yu, 和 Xiaohua Jia。2025 年。《大语言模型能否拒绝回答它们不知道的问题？在事实性任务中衡量知识感知的拒绝能力》。arXiv 预印本 arXiv:2510.01782。
- Shao 等人 (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, 和 Daya Guo。2024 年。《DeepSeekMath：推动开放语言模型数学推理的极限》。arXiv 预印本 arXiv:2402.03300。
- Suzgun 等人 (2025) Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, 和 James Zou。2025 年。《动态速查表：基于自适应记忆的测试时学习》。arXiv 预印本 arXiv:2504.07952。
- Terminal-Bench 团队 (2025) Terminal-Bench 团队。2025 年。《Terminal-Bench 2：在困难、真实的命令行任务上对智能体进行基准测试》。https://www.tbench.ai/。引用信息待定。
- Wang 等人 (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, 和 Denny Zhou。2022 年。《自一致性改进了语言模型中的思维链推理》。arXiv 预印本 arXiv:2203.11171。
- Xiao 等人 (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, 和 Jian-Yun Nie。2023 年。《C-Pack：通用中文嵌入向量的打包资源》。arXiv 预印本 arXiv:2309.07597。
- Yang 等人 (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, 和 Xinyun Chen。2023 年。《大语言模型作为优化器》。arXiv 预印本 arXiv:2309.03409。
- Yuksekgonul 等人 (2024) Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, 和 James Zou。2024 年。《TextGrad：通过文本实现自动“微分”》。arXiv 预印本 arXiv:2406.07496。
- Zhou 等人 (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, 和 Jimmy Ba。2022 年。《大语言模型是人类级别的提示词工程师》。arXiv 预印本 arXiv:2211.01910。
附录 A 与相关工作的比较
| 满足 RHO 的设置 | ||||||
| 方法 | 工具链架构 | 反馈信号 | 成本 | 无标签 | 完整工具链 | 单次通过 |
| 验证反馈优化 | ||||||
| OPRO (Yang et al., 2023) | 提示词 | 验证指标 | 迭代搜索 | ○ | ○ | ○ |
| DSPy (Khattab et al., 2023) | 提示词示例 | 验证指标 | 迭代搜索 | ○ | ○ | ○ |
| TextGrad (Yuksekgonul et al., 2024) | 提示词 | 文本梯度 | 迭代搜索 | ○ | ○ | ○ |
| GEPA (Agrawal et al., 2025) | 提示词 | 验证指标反思 | 迭代（遗传算法） | ○ | ○ | ○ |
| ADAS (Hu et al., 2024) | 完整智能体代码 | 验证准确率 | 25–30 次迭代 | ○ | ● | ○ |
| Meta-Harness (Lee et al., 2026) | 完整工具链代码 | 搜索集得分 | 20 次迭代 | ○ | ● | ○ |
| 基于经验的自我改进 | ||||||
| 动态速查表 (Suzgun et al., 2025) | 速查表文本 | 自我判断 | 在线流 | ● | ◐ | ○ |
| ReasoningBank (Ouyang et al., 2025) | 记忆项 | 大语言模型作为评判者 | 在线流 | ● | ○ | ○ |
| MemMA (Lin et al., 2026) | 记忆条目 | 合成探针问答 | 在线（会话内） | ● | ○ | ○ |
| 睡眠时间计算 (Lin et al., 2025) | 输入上下文 | 无 | 预计算 | ● | ○ | ● |
| SkillOS† (Ouyang et al., 2026) | 技能列表 | 强化学习奖励† | 强化学习训练 | ◐ | ◐ | ○ |
| RHO（我们的方法） | 工具技能指令 | 自我偏好 | 单次通过 | ● | ● | ● |
†SkillOS 的强化学习奖励是一种大语言模型作为评判者的自我信号，而非真实标签，但它仍然需要带标签的训练语料库、一个独立的评判模型以及权重更新。
表5将RHO与第2节讨论的先前方法进行了比较。对于每种方法，我们报告了其编辑的框架表面、驱动它的反馈信号以及它产生的成本模式，并标记了它是否在三个标准上满足RHO的设置，即优化是否无标签、是否编辑完整框架、以及是否作为单次离线回顾性过程运行。验证反馈优化器达到了框架表面轴的完整框架端，其中ADAS和Meta-Harness重写了可执行代码，但该系列的每个成员都使用真实指标引导搜索并对其进行迭代。基于经验的方法去掉了标签，但它们仅在线整理记忆或技能文本，而让可执行框架保持不变，同时睡眠时间计算离线预计算上下文但不进行任何优化。因此，每种先前方法最多满足三个轴中的两个，而RHO是唯一同时满足所有三个轴的方法。
附录B 提示词
本附录收集了实例化RHO五个智能体操作符（第3节，算法1，图2）的提示词，即核心集选择中使用的难度判断器、诊断分析、以及。提示词逐字复制。{name}形式的占位符在调用时用每个代码块下描述的值填充。
B.1 求解
每次调用都会将框架和任务物化到harness/和task/下的新工作空间中，并将以下包装指令交给智能体。该包装器与任务无关，因为框架目录包含所有任务塑造指导，智能体仅被告知如何读取工作空间以及如何提供最终答案。相同的包装器用于基线运行和每个候选框架运行，因此是唯一变化的输入。
B.2 核心集选择（难度判断器）
难度评判器会生成分数和抽象指纹，这些数据驱动了第4.1节中描述的DPP核心集选择器。它接收任务描述，以及当前测试框架下一条较短历史轨迹的长度受限摘要。该摘要会被截断头部和尾部，以符合固定的token预算，并且在摘要展示给评判器之前，任何读取任务预期答案文件的命令都会被清除。评判器被要求使用与任务无关的结构化词汇来保留指纹，这样来自不同代码库的指纹在余弦相似度下仍具有可比性。
{query}
是自然语言的任务描述。
{trajectory_digest}
是同一任务的一条历史轨迹经过清除、截断头部和尾部后的摘要。这个难度值就是进入DPP核函数的分数，并按第4.1节所述进行归一化处理，而指纹则被嵌入为一个单位向量，用于定义相似度矩阵。
B.3 诊断
诊断提示实现了（第4.2节）。工作区向智能体提供原始任务、每次运行所使用的共享测试框架，以及运行目录。智能体执行一个五步工作流，包括：逐条轨迹检查、失败模式分析（自验证）、跨轨迹不一致性分析（自一致性）、一个单一的高层级测试框架改进方向，以及一个严重性分数——该分数在下游流程中同时充当软注意力权重。结构化的JSON输出将每个字段绑定到固定槽位，以便能够根据严重性进行注意力分配。
轨迹目录包含完整的事件流、智能体的最终消息，以及运行产生的任何工作区差异。
自验证
自一致性
表
中的消融实验移除了相应的步骤和相应的输出字段，而周围的框架结构保持不变。
B.4 优化
优化提示词实现了（§4.3）。编辑器智能体被授予对测试框架新副本以及按严重程度排序的每个任务诊断目录的写入权限。指令将严重程度设定为软注意力权重，要求在进行任何编辑前进行跨任务模式匹配，并阻止针对特定任务的硬编码修复。每个候选方案均使用相同提示词独立采样，底层智能体的随机性提供了多样性。
每个
diagnoses/task_XXXX/
子目录包含以 Markdown 格式呈现的序列化诊断 JSON，以及原始任务提示词。子目录按严重程度降序索引，以便智能体首先遇到最关键的诊断。
B.5 成对排序
排序提示词实现了最佳接受策略中使用的功能（§4.3 和算法 1）。评估者看到任务、两个测试框架以及两条轨迹。候选轨迹显示为 trajectory_A，基线轨迹显示为 trajectory_B，编排器会对返回的整数取反，使得标量分数始终以基线候选方案为基准，无论呈现顺序如何。将候选方案放在前面可以减少我们在试运行中观察到的后选项偏好偏差。下面的评分标准使用整数标度，下游仅使用分数的符号和相对大小（§3）。
算法 1 中的候选分数
在核心集上对该整数（经过上述方向翻转）取平均值，只有当 时候选方案才被接受，否则测试框架保持为 。
附录 C 超参数与基础设施
表 6 列出了我们实验中使用的所有超参数和基础设施设置。为方便起见，主文（第 3 节和第 5.1 节）中已报告的值在此重复列出，之前未说明的值在此引入。其目的是提供一份可供审计的规范，每列对应一个参数，每个单元格对应一个值，不隐含任何默认值。求解器运行时间和评分方面的数据集特定覆盖规则推迟到附录 E 中说明，每个智能体角色的提示词文本见附录 B。
| 参数 | 取值 | 描述 |
| 骨干智能体 | ||
| 模型† | Codex gpt-5.5 | 在求解、优化和排序中共享。 |
| 推理投入† | 高 | 统一应用于所有角色。 |
| 采样温度 | 提供商默认值 | Codex推理模型不暴露温度参数。 |
| 提供商 | 云端托管的gpt-5.5（Codex CLI） | 单一托管端点。 |
| 核心集选择 | ||
| 选择器 | DPP，贪心MAP | Kulesza和Taskar（2012）。 |
| 核心集大小† | 10 | 训练任务。 |
| DPP权重† | 0.7 | 难度/多样性权衡。 |
| 得分下限 | 0.1 | 归一化难度的下界。 |
| 评判模型 | Codex gpt-5.5，高推理投入 | 与求解器使用相同的骨干模型。 |
| 轨迹摘要预算 | 10,000个BPE模型token，头尾截断 | 已清除暴露真实答案的命令。 |
| 指纹嵌入向量 | BAAI/bge-large-en-v1.5，1024维 | 本地执行（Xiao等人，2023）。 |
| 组式运行 | ||
| 每个任务的运行次数 | 3 | 每个核心集任务下的并行求解。 |
| 求解的挂钟超时时间 | 900秒 | 数据集覆盖见附录E。 |
| 诊断提示词 | 见附录B | 自验证自一致性线索。 |
| 严重程度范围 | 软注意力权重，非硬性阈值。 | |
| 最佳方案生成提案 | ||
| 候选方案† | 3 | 并行优化器样本。 |
| 优化器提示词 | 见附录B | 以 为条件。 |
| 排序超时时间 | 每次成对调用300秒 | 每个 。 |
| 接受阈值 | （严格） | 核心集上的平均成对得分。 |
| 基础设施 | ||
| 智能体调用并发数 | 10个并发调用（上限30） | 表1运行的并行度。 |
| 每轮实验次数 | 1 | 除非另有说明。 |
| 持久化 | 完整日志保存至磁盘 | 提示词、补全结果、轨迹、诊断信息、 |
| 候选方案、差异对比、得分、运行摘要。 | ||
表 6 中的若干选择需要简要说明。我们将 rollout 次数设为三次，因为三次 rollout 足以让诊断提示词浮现出跨轨迹的不一致性，而更大的 rollout 次数在试点中只会线性增加成本，却不会增强诊断信号。我们对 DPP 进行加权，使得难度项主导多样性项，其原则是：一个未被选中的简单任务所携带的优化信号，还不如一个冗余的困难任务。具体来说，难度项通过难度权重进入，而系数 2 用于抵消难度项在 中重复出现的影响，从而使 和 直接对难度项和多样性项进行加权。我们将接受门限设为严格正数而非非负数，因为成对自偏好是一种有噪声的估计器，而如果打破平局时倾向于变更，则会在无预期收益的情况下增加回归风险。最后，我们对求解器、优化器和排序器使用相同的 Codex gpt-5.5 骨干模型，因为如果将求解器与评判器解耦，就会引入一个混淆因素，使得测得的保留集增益可能归因于更强的评判器，而非 RHO 本身。上述按数据集覆盖的配置覆盖项见附录 E，各基线特定的配置见附录 F。
附录 D 流水线实现细节
本附录记录了决定 RHO 行为但未在第 4 节或算法 1 中出现的各项设计选择。方程和算子签名已在正文中固定，以下内容则是使其可执行的实现机制。
D.1 测试框架表示与挂载
工具包是一个没有固定模式的目录文件集合，其中文字指令、可执行脚本和结构化配置并存。在每次算子调用时，工具包会被实例化为智能体工作目录的一个子目录，智能体读取它的方式与读取任务文件相同，而不是作为系统提示词的前缀。算子按照惯例以只读方式挂载工具包，而则挂载一个具有写入权限的新副本，并接受智能体在文件系统上留下的任何内容。工具包通过内容进行比较，因此如果返回的工具包与输入完全相同，则该候选方案被视为无操作，在评估前被丢弃。这使工具包界面保持与工具无关，并允许优化器在不改变表示形式的情况下扩展这三种文件类型中的任何一种。
D.2 角色分离与工作空间隔离
同一个主干网络执行、、诊断分析（/）、以及。角色分离通过工作空间内容实现，而非改变主干网络，每个算子都在一个仅包含其应见输入的全新工作空间中运行。看到任务文件和当前工具包。诊断分析看到任务及其对应的轨迹组。看到工具包目录、诊断指令以及触发这些指令的轨迹展开结果。最后，同时看到任务、两条轨迹以及两个候选工具包目录。因此，按角色进行消融相当于交换提示词和输入，而非交换模型，这使得各算子之间具有直接可比性。
D.3 成对排序中的顺序、解析与失败处理
每个（任务，候选方案）对调用一次。我们首先呈现候选轨迹，其次呈现基线轨迹，然后对解析出的标量值取反。这种交换是成对评判器中针对位置偏差的标准缓解措施，无需每次调用时随机化顺序。评判器被限制为返回一个范围内的单个整数并附带一句理由说明，且我们不进行重试。任何解析或执行失败都确定性地返回零。这使得成为一个严格的悲观主义者，因为静默失败会将均值拉向拒绝阈值，而非远离它。
D.4 诊断与排序输入
诊断分析会消耗单个任务的全部 rollout 结果，并生成一条带有严重性权重的单一文本指令，而该过程严格来说是成对进行的。我们不会将一组结果压缩成 Bradley–Terry 评分用于评估。对于每个候选方案，我们会将该候选方案在某个任务上的 rollout 结果与从原始组中抽取的一条固定基线轨迹进行配对，并且我们在所有候选方案中保持该基线不变，从而使比较始终锚定在同一参考点上。其结果是，该过程仅作为诊断工具进入流程，因为它通过跨轨迹的不一致性来强化指令，但并不对哪个候选方案胜出进行投票。
D.5 优化器动作空间
这是一个代码智能体的调用，而非受限的文本重写器。它将物化后的测试框架视为一个文件系统，并可以添加、删除或修改其中的任何文件。我们按严重性降序排列诊断指令，并将严重性作为软注意力权重（而非硬优先级队列）暴露给智能体，这样智能体仍然可以合并或跳过存在冲突的建议。我们不会从智能体的最终消息中解析出差异补丁，当调用返回时，新的测试框架状态直接从目录本身恢复。这保持了动作空间与表示形式的一致性，即任何适合放在目录中的内容都是有效的编辑。
D.6 接受门控与空操作
只有当最佳候选方案在核心集上的平均成对评分严格为正时，更新才会被接受，平均值为零的更新则被拒绝。在门控之前，如果优化器失败、调用超时或生成的测试框架与输入完全相同，则该候选方案会被丢弃。如果所有候选方案都被丢弃，或者所有幸存的候选方案评分均为零，RHO 将保持测试框架不变。这种不更新的行为是结果的一部分，而非工具故障。在图 5 中，那些核心集未能暴露有用弱点（纯覆盖率、纯难度）的选择器，其保留的准确率接近 Vanilla Codex 基线。
D.7 持久化
我们持久化存储输入测试框架 ID，以及每个算子产生的轨迹（即诊断分析、 和 ，通过 ID 标识）。我们还进一步持久化存储诊断指令、候选测试框架 ID 及其与输入的差异、每个候选对的成对分数、平均成对分数以及接受标志。每条存储的轨迹都包含其完整的事件流、最终消息、工作区差异和挂钟时间。这些记录使得无需重新运行智能体即可进行下游审计、重新评分和消融实验成为可能。
附录 E 数据集规范
本附录记录了我们从每个上游基准测试中获取的内容、如何划分数据、智能体在求解时看到的内容以及评分如何运行。固定的上游提交版本使得从干净的代码检出即可重现数据划分和评分器。
E.1 SWE-Bench Pro
SWE-Bench Pro（Deng 等人，2025）是我们套件中的长周期软件工程基准测试。任务只有在多文件补丁通过上游测试集时才视为解决，因此失败模式是具体的，并且可以追溯到测试框架的仓库约定和构建命令。
来源。我们从 Hugging Face 加载 ScaleAI/SWE-bench_Pro 的测试集划分。评估器脚本和每个实例的 Docker 镜像来自官方 scaleapi/SWE-bench_Pro-os 仓库，并固定到特定的提交版本。111提交版本 0c64e26f00b9c190432de7fc520c8ceed5c25518。
划分。行按照（种子，实例 ID）的 SHA-256 哈希值排序，种子固定。前 100 行构成训练池，接下来的 100 行构成保留测试池，其余行不使用。我们报告的是保留测试池的结果。
求解接口。每个任务会生成一个描述问题的提示词文件，以及一份基于其基础提交版本的上游仓库的新克隆，挂载在工作区中。智能体直接编辑仓库文件，不注入任何额外工具。
评分。我们通过将智能体的工作区编辑重新应用到一份新的代码检出上，并执行 `git diff --binary` 来提取智能体的补丁。如果工作区路径失败，我们会回退到扫描智能体的最后一条消息，查找其中被围栏包裹的差异代码块。在评分前，会剔除二进制块以及自动生成的路径（依赖、缓存和构建目录）。随后，该补丁会在 SWE-Bench Pro 作者分发的官方按实例划分的 Docker 镜像内应用，并调用官方的运行和解析脚本。只有当每个测试用例都正确通过时，任务才算成功。每个任务的 Docker 挂钟时间预算为一小时。
E.2 Terminal-Bench 2
Terminal-Bench 2（Terminal-Bench 团队，2025）包含可执行的命令行任务。失败意味着错过奖励，而非违反约定，因此测试框架主要保证了智能体在检查状态和串联命令时的一致性。
来源。我们使用上游 Terminal-Bench 2 仓库的固定提交版本222提交号 53ff2b87d621bdb97b455671f2bd9728b7d86c11.，共得到 89 个任务。
划分。任务按照与 SWE-Bench Pro 相同的种子方案进行哈希排序。前 30 个构成训练池，其余 59 个构成保留池。
求解接口。每个任务会启动一个全新的 Docker 容器，其镜像由任务声明。容器带有清理标签以及每个任务的 CPU 和内存限制。智能体的主机工作区会以绑定挂载的方式放入容器内。求解提示词指示智能体在主机上编写 shell 脚本，并在容器内执行这些脚本。当任务声明的智能体超时时间到达时，一个挂钟看门狗会终止该容器。
评分。我们在仍在运行的容器内，于任务声明的验证器超时时间内，运行该任务的上游测试套件。验证器会将 0 或 1 的奖励写入一个已知路径。当且仅当奖励等于 1 时，任务才算通过。我们不应用难度过滤器，因此保留池的数值涵盖了上游的难度分布。
数据完整性。求解提示词指示智能体不得读取测试目录、上游解决方案或验证器日志。这是一个提示词层面的约定，而非沙箱限制。我们将其记录在案以保证透明度，因为一个足够对抗性的智能体可能会读取验证器并操纵奖励。我们依赖保留池以及表1中的跨基准一致性来检测此类行为，且尚未观察到这种情况。
E.3 GAIA-2
GAIA-2（Froger 等人，2026年）与两个编程基准的不同之处在于，其环境独立于智能体而演化。因此，测试框架层必须对部分可观测和异步事件下的行为进行编码，而不仅仅是稳定的构建约定。
来源。我们从 Hugging Face 上加载 GAIA-2 发布版本中 mini 配置下的验证集，共获得 200 个场景。每个场景都包含一个异步事件流和一个上游写入操作验证器。
划分。场景使用相同的种子方案按哈希值排序。前 100 个构成训练池，其余 100 个构成保留池。我们报告的是保留池部分的结果。
求解接口。每个场景的工作区包含一个任务提示词、一个工具调度器和一个工具目录。智能体通过每次调用的单个 shell 命令来调用工具。调度器将每次调用转发给一个运行上游环境的边车进程，该进程推进模拟时间并重放预定事件。场景的真实状态完全由边车进程持有，智能体无法读取。
评分。当智能体完成时，边车进程会调用该场景的上游验证器。评判 LLM 与我们其他地方使用的 Codex gpt-5.5 骨干模型相同，路由默认指向同一个 Azure Foundry 端点。当且仅当上游验证器报告成功时，该场景才算通过。
环境修改。上游环境有两项变更会影响报告数据，必须予以说明。首先，每轮对话中 `send_message_to_user` 的调用次数上限从一次提升至四次，因为原有上限会惩罚那些本可正确完成任务但回复冗长的智能体。其次，三项可选的评估器宽松开关（事件过滤、宽松的逐应用 UI 评估、以及琐碎文件系统读取过滤）可作为消融实验的变量使用，但在表 1 报告的实验中均处于禁用状态。
附录 F 基线实现
所有基线均在相同的 Codex gpt-5.5 主干模型上以高推理强度运行。仅轨迹类方法（动态速查表、推理库、睡眠计算）共享 RHO 的核心集选择预算，即 10 个训练任务和 3 个候选任务。各基线的差异仅在于离线阶段持久化存储的内容以及求解器使用这些内容的方式。本附录将逐一说明。
F.1 动态速查表（Suzgun 等人，2025）
动态速查表会整理一份从过往轨迹中提取的可复用事实与流程的运行列表。
持久化层。一个 Markdown 文件存在于测试框架内部，其结构为 `<memory_item>` 块，包含描述、工作示例和使用次数。
离线阶段。一个策展智能体按顺序遍历选定的训练任务。对于每个任务，它会读取当前的速查表以及求解记录，然后就地重写速查表。
在线使用。速查表是测试框架的一部分，求解器在常规框架查找过程中会读取它。不会注入额外的提示词前缀。
忠实性。上游的动态速查表会在每个任务完成后更新速查表，因此下一个任务的求解器能看到更新。我们的默认配置是先对所有选定的训练任务执行求解，然后再调用策展智能体，最后提交一份速查表。与上游逐示例更新相匹配的单任务配置可作为敏感性设置使用，而表 1 使用的是默认配置。
F.2 推理库（Ouyang 等人，2025）
推理库存储从过往轨迹中提取的可复用推理模式，并在推理时按需检索使用。
持久化层。一个 JSONL 格式的条目库，每个条目包含标题、描述、正文以及预计算的嵌入向量。该条目库位于 harness 目录之外，因为求解时的访问是通过相似性检索完成的，而非通过 harness 的具体化。
离线阶段。对于每个选定的训练任务，我们求解该任务，让评判器判断轨迹是否成功，然后运行一个提取提示词，从轨迹中提炼出可复用的推理模式。提取出的条目被追加到条目库中，并在本地计算嵌入向量。
在线消费阶段。在每次保留的求解过程中，我们使用相同的编码器对任务描述进行编码，通过余弦相似度检索出 top- 个条目，并将它们渲染为一段记忆前缀，附加到求解指令之前。top- 固定为 1。
保真度。我们将上游的 Gemini 嵌入向量替换为 BAAI/bge-large-en-v1.5（Xiao 等人，2023），这是一个 1024 维的本地执行句子编码器，这样所有使用检索的基线方法都共享相同的编码器、维度和存储后端。这消除了记忆库质量与嵌入提供者质量之间的混淆因素。
F.3 休眠时间计算（Lin 等人，2025）
休眠时间计算将过去的轨迹预处理为紧凑的笔记，在推理时附加到智能体的上下文中，遵循 Letta 的实现方式（Packer 等人，2023）。
持久化层。一组存储在 harness 内部的有界 Markdown 记忆块，通过结构化的编辑工具（插入、替换、重新思考、完成）进行修改，这些工具镜像了上游 Letta 的语义。
离线阶段。一个休眠时间智能体遍历选定的训练任务。对于每个任务，它读取任务的提示词和轨迹，然后发出一系列记忆编辑工具调用，以更新 harness 的记忆块。系统提示词是上游 Letta 的系统提示词，逐字使用。
在线消费阶段。求解器将记忆块作为 harness 的一部分进行读取，就像读取任何 harness 文件一样。
范围。编辑操作在各个任务之间累积，休眠时间智能体针对一个不断演变的单一记忆状态，一个任务接一个任务地工作。我们不会在训练任务之间重置记忆。
F.4 元 Harness（Lee 等人，2026）
Meta-Harness 是验证反馈的参考基准，其中元智能体重写测试框架，并根据标注的验证集对重写结果进行评分。
持久化层。一系列候选测试框架目录，配有一个搜索历史记录，记录每个候选方案的验证通过率。
离线阶段。一个外层循环交替进行以下操作：提议者读取搜索历史并生成新的测试框架，评估者使用数据集的地面真值评分器对提议的测试框架在搜索任务集上进行评分。Upstream 的默认预算为 20 次外层迭代、每次迭代 3 个候选方案、每个任务 2 次求解尝试，远高于我们分配给 RHO 的预算。
验证反馈足迹。提议者从历史记录中读取每个候选方案的每个任务得分、平均得分和通过率，评分器的信号直接塑造下一个提议。这是验证反馈系列的特征，也是 Meta-Harness 与仅基于轨迹的基线方法无法比较的维度。
匹配预算配置。对于表 2，我们运行外层循环一次迭代，包含 3 个候选方案和每个任务 1 次求解尝试，因此候选方案数量与 RHO 匹配。表 2 报告的是验证评分的成本。
共享配置。所有四个基线方法和 RHO 都从给定数据集的相同空测试框架开始，使用相同的选定训练任务核心集，并使用相同的评分器在相同的保留测试集上进行评分。唯一有意的变化维度是离线阶段持久化什么以及求解器如何消费它。Meta-Harness 额外消费了验证评分，这是表 2 衡量的维度。
| 方法 | rollout | diagnose | optimize | after | rank | test | 总计 | llm |
| Vanilla Codex | 0 | 0 | 0 | 0 | 0 | 100 | 100 | 0 |
| ReasoningBank (Ouyang et al., 2025) | 10 | 0 | 0 | 0 | 0 | 100 | 110 | 20 |
| 动态速查表（Suzgun 等人，2025） | 30 | 0 | 10 | 10 | 10 | 100 | 160 | 0 |
| 睡眠时间计算（Lin 等人，2025） | 30 | 0 | 30 | 30 | 30 | 100 | 220 | 0 |
| RHO | 30 | 10 | 3 | 30 | 30 | 100 | 203 | 0 |
附录 G 优化阶段计算成本
本附录记录了 RHO 以及表 1 和表 2 中报告的基线方法的优化阶段计算成本。我们报告这些数字，并不声称具有效率优势。在匹配的核集预算（, ）下，仅使用轨迹的方法中，RHO 与睡眠时间计算和动态速查表处于同一数量级，且比 ReasoningBank 更昂贵。匹配预算的 Meta-Harness 配置是唯一成本在性质上有所不同的基线，并且其上游默认预算会更大，我们将在末尾讨论这一点。
G.1 核算范围
每种方法在离线阶段都会调度两种类型的模型调用，再加上评估时每个任务的求解调用。我们分别统计它们，因为它们通过不同的后端运行，且单位成本不同。
Codex（智能体）调用。一个运行至完成的 Codex CLI 子进程计为一次智能体调用。每次调用都会持久化一个轨迹目录，记录角色（solve、diagnose、optimize、rank 之一）、测试平台 ID、任务 ID、挂钟时长和退出状态。我们通过统计这些目录来获得每种方法的总数。
辅助 LLM 调用。对同一 gpt-5.5 主干模型发起的单次直接 API 调用，在 Codex CLI 外部执行。只有 ReasoningBank 在离线阶段使用这些调用，即每个训练任务两次，用于成功判断和记忆项提取。核集选择在任何方法开始之前，也会对每个候选任务发起一次辅助 LLM 调用。此成本在匹配核集的方法之间共享，并在本附录末尾报告一次。
嵌入向量调用。ReasoningBank 和 DPP 核集选择都调用 BAAI/bge-large-en-v1.5（Xiao 等人，2023）进行指纹或查询嵌入。我们在本地执行编码器，因此这些调用不会访问远程端点，也不与 LLM 总数合并。
G.2 各方法分解
表 7 将 SWE-Bench Pro（训练任务、保留任务）上的优化阶段按角色分解为智能体调用。Terminal-Bench 2 仅在与 SWE-Bench Pro 匹配的 GAIA-2 上有所不同。
该表是对每种方法如何分配离线预算的精确记录。ReasoningBank 为每个训练任务执行一次求解，且从不提出候选工具框架，因此离线阶段仅在保留评估之上产生额外的 Codex 调用成本，其辅助 LLM 调用包括成功判定器和记忆项提取器。Dynamic Cheatsheet 和 Sleep-time Compute 均对每个训练任务进行多次展开以呈现方差，随后运行一个优化器智能体。Dynamic Cheatsheet 对所有任务调用一个单一的策展器（候选方案），而 Sleep-time Compute 每次重启对每个任务执行一次编辑（3 次重启 × 任务数 = 优化调用数）。两者随后将每个唯一的候选工具框架提交给与 RHO 共享的求解后步骤和成对排序步骤。RHO 的占用空间与 Sleep-time Compute 的区别仅在于：将每个任务的优化调用替换为诊断调用，加上最优提议器中的候选级优化调用，而求解后和排序阶段在候选方案数量上相同。
G.3 挂钟时间
表 8 报告了每个（方法，数据集）单元格的两种互补挂钟时间测量值。前者是运行中每次智能体调用的挂钟时间之和，即在一个智能体进程上串行执行相同工作负载的成本。端到端时间是从运行开始到完成所经过的时间，基于我们 10 个并发智能体调用和 10 个并发评分器的基础设施设置。两列之间的比值反映了工作负载成功并行化的程度，其中纯轨迹方法达到约 10 倍加速，而 ReasoningBank 达到约 2 倍加速，因为其离线阶段在结构上是顺序的（每个训练任务的求解必须在下一个任务的检索之前提交到记忆）。Meta-Harness 是一种验证反馈方法而非纯轨迹方法，因此我们将其预算单独报告在 §G.4 中，而非此处。
| SWE-Bench Pro | Terminal-Bench 2 | GAIA-2 | ||||
| 方法 | 端到端 | 端到端 | 端到端 | |||
| 基础版 Codex | 9.4 | 1.1 | 4.3 | 0.6 | 4.2 | 0.6 |
| ReasoningBank (Ouyang et al., 2025) | 17.5 | 4.6 | 8.0 | 2.4 | 7.2 | 2.0 |
| Dynamic Cheatsheet (Suzgun et al., 2025) | 17.9 | 2.5 | 10.8 | 1.9 | 6.5 | 1.2 |
| Sleep-time Compute (Lin et al., 2025) | 22.2 | 3.2 | 14.5 | 2.2 | 7.0 | 1.2 |
| RHO | 23.1 | 3.2 | 15.5 | 2.5 | 9.2 | 1.4 |
有两点观察值得明确指出。首先，单次调用成本在不同数据集之间差异很大（GAIA-2 平均每次调用约 秒，Terminal-Bench 2 约 秒，SWE-Bench Pro 约 秒），因此智能体调用次数仅在单个数据集内具有参考价值，不能仅通过累加调用次数来跨数据集比较成本。其次，在匹配的核心集和 条件下，RHO 在每个数据集上的成本均与 Sleep-time Compute 相差在 以内，比 Dynamic Cheatsheet（其使用 ）更昂贵，并且显著高于 ReasoningBank（后者不进行分组展开、不生成候选方案、不进行成对排序）。
G.4 匹配预算元测试框架
Meta-Harness 的计算开销在数量级上更大，因为验证评分会反馈给提议者。在其上游默认配置下（外层迭代次数、每轮候选数、每任务求解尝试次数），仅离线阶段在 SWE-Bench Pro 上就需要消耗 次智能体调用——几乎是 RHO 优化阶段调用次数（ 次）的十二倍。我们在表 2 中报告的预算匹配配置将其降低至 、 、 ，优化阶段调用次数为 次；该配置在 SWE-Bench Pro 上的保留集通过率（ ）与 RHO 的结果一并呈现在正文中。同一张表中报告的 10 轮配置保留了 ，但将 提高至 ，优化阶段调用次数为 次（RHO 优化阶段调用次数为 次）。这些计数不包括共享的保留集评估，无论采用何种优化器，每种方法都只需支付一次该评估成本。由于 Meta-Harness 是一种基于验证反馈而非仅轨迹的方法，因此未纳入表 8 的挂钟时间对比。
G.5 核心集选择成本（共享）
核心集选择在任何方法的离线阶段开始前运行一次。对于 SWE-Bench Pro，这需要消耗 次辅助大语言模型调用（每个候选任务进行一次难度判断），外加一次对相同指纹的批量嵌入向量调用，之后所选任务 ID 将被持久化，并由每个基线方法原样复用。该成本每个数据集仅支付一次，不归属于任何单个方法。
附录 H 优化后的 Harness 工件
本附录逐字复现了 RHO 在每个基准测试上产生的最高分 harness 的完整内容，即图 3 中总结的 harness。每个 harness 是一个包含 Markdown 格式的指令和技能文件以及可执行工具脚本的目录；每个文件均完整展示，无任何删节。文件按基准测试分组，并标注了其在 harness 目录中的路径及其角色（指令、技能或工具）。
H.1 SWE-Bench Pro
H.2 Terminal-Bench 2
H.3 GAIA-2
Evolving Agents in the Dark:
Retrospective Harness Optimization via Self-Preference
Wenbo Pan
Shujie Liu
Chin-Yew Lin
Jingying Zeng
Xianfeng Tang
Xiangyang Zhou
Yan Lu
Xiaohua Jia
City University of Hong Kong
Microsoft Research Asia
Abstract
AI agents rely on a harness of skills, tools, and workflows to solve complex problems. Continually improving this harness is essential for adapting to new tasks. However, existing optimization methods typically require ground-truth validation sets, yet such labeled data is difficult to acquire in practical deployment settings. To address this problem, we introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimizes the agent harness using only past trajectories. Specifically, RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel. The agent analyzes these rollouts using self-validation and self-consistency, then generates candidate harness updates and selects the most effective one by its own pairwise self-preference. We evaluate RHO across three diverse domains, spanning software engineering, technical work, and knowledge work. Notably, a single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading. Furthermore, our analysis demonstrates that RHO effectively targets prior failure modes. As a result, the optimized harness alters the agent’s behavior patterns and sustains higher accuracy during long-horizon sessions. Code is available at https://github.com/wbopan/retro-harness and the project website at https://paper-rho.wenbo.io.
Evolving Agents in the Dark:
Retrospective Harness Optimization via Self-Preference
Wenbo Pan1 Shujie Liu2 Chin-Yew Lin2 Jingying Zeng2 Xianfeng Tang2 Xiangyang Zhou2 Yan Lu2 Xiaohua Jia1 1City University of Hong Kong 2Microsoft Research Asia
1 Introduction
A harness enables an AI agent to complete complex tasks by providing it with available skills, workflows, and tools. One important research question is how to improve the harness continuously. Specifically, after an agent is deployed, we aim to continually evolve its harness by learning from past experiences, which in turn improves its performance on future tasks.
Prior work has proposed various methods for evolving the agent harness (Zhou et al., 2022; Yang et al., 2023; Khattab et al., 2023; Yuksekgonul et al., 2024; Agrawal et al., 2025; Hu et al., 2024; Lee et al., 2026). However, these methods rely on scoring against a validation set to guide the improvements. In practical deployment scenarios, it is often difficult to collect a validation set that accurately estimates the distribution of future tasks to validate the updated harness. On the other hand, the continuous operation of an agent naturally produces a rich set of trajectories from past tasks. This leads to our central question. Can we improve the agent harness to enhance future performance when we only have access to past trajectories?
To address this problem, we propose Retrospective Harness Optimization (RHO), a self-supervised method that optimizes the harness through a retrospective analysis of past trajectories. This method employs the agent’s internal self-preference over trajectories to guide the optimization process. Figure 1 contrasts RHO with conventional validation-feedback optimization, which iterates against a labeled validation set.
Figure 2 illustrates this process. Specifically, given a large set of past trajectories, RHO first selects a diverse and challenging coreset of tasks. Then the agent re-attempts each task in the coreset multiple times to generate parallel trajectories. Building on this, we extract two diagnostic signals, namely self-validation within a trajectory and self-consistency across parallel trajectories. These signals are then used to instruct the generation of harness updates. Finally, by using the agent’s pairwise self-preference, we select the most promising harness from the newly generated proposals.
We evaluate the effectiveness of RHO across three agent domains that span software engineering, technical work, and knowledge work. RHO consistently improves the agent’s performance across all three domains. Notably, by running a single round of retrospective harness optimization on software-engineering trajectories, we improve the pass rate on SWE-Bench Pro (Deng et al., 2025) from 59% to 78%, without depending on grading against a validation set.
Furthermore, we provide a detailed analysis on how the retrospective optimization process improves performance. We observe that RHO designs specific skills and tools targeting typical failure modes encountered in past tasks. These components reshape the agent’s action patterns and help it sustain higher accuracy in long-horizon sessions. Additionally, we quantitatively analyze the contributions of the diagnostic signals during the retrospective process. This analysis demonstrates that each step in RHO progressively isolates signals that contribute to performance improvements.
-
We propose retrospective harness optimization, which addresses the gap of improving the full harness (including memory, context, skills, and tools) exclusively from unlabeled trajectories.
-
We evaluate RHO across three scenarios and show that retrospective analysis consistently outperforms straightforward experience accumulation and surpasses validation-feedback-driven evolution under a comparable budget.
-
We provide a quantitative analysis of the impact of harness optimization on agent performance, showing that gathering effective improvement signals leads to targeted changes in the harness and optimizes the agent’s behavior.
2 Related Work
Harness optimization.
Harness optimization improves an agent by editing the prompts, program parameters, or workflow code that surround a fixed model. One line optimizes prompts or pipeline parameters against a labeled metric, spanning LLM-as-optimizer search (Yang et al., 2023), declarative pipeline compilation (Khattab et al., 2023), textual-gradient updates (Yuksekgonul et al., 2024), and reflective prompt evolution (Agrawal et al., 2025). A more agentic line lets a meta-agent rewrite the agent’s own code, where ADAS searches the space of agentic system designs (Hu et al., 2024) and Meta-Harness searches over harness code using the execution traces and scores of prior candidates (Lee et al., 2026). Although these methods differ in the surface they edit, all of them steer the search with a labeled validation metric. RHO departs from this paradigm, requiring no validation feedback and improving the harness in a single retrospective pass over unlabeled past trajectories.
Agent self-improvement.
A second line improves agents from their own past experience, using the agent’s self-judgment over trajectories in place of ground-truth labels. Dynamic Cheatsheet maintains a self-curated memory of reusable strategies and code snippets at test time (Suzgun et al., 2025), while ReasoningBank distills generalizable reasoning strategies from self-judged successes and failures (Ouyang et al., 2025). MemMA coordinates the memory cycle with multiple agents and repairs its memory bank against self-generated probe questions (Lin et al., 2026), Sleep-time Compute precomputes useful context offline before queries arrive (Lin et al., 2025), and M⋆ evolves the memory system itself as an executable program, discovering a task-specific memory harness per task family (Pan et al., 2026a). Concurrent to our work, SkillOS instead trains a skill curator with reinforcement learning from outcome and judge rewards, updating a skill repository from accumulated experience (Ouyang et al., 2026). These methods enrich an agent’s stored memory, context, or skill list while leaving the rest of the harness untouched. RHO instead optimizes the full harness, including executable tools and instructions, rather than memory alone. Appendix A gives a detailed comparison with related work.
3 Problem Setting
We define a harness as a persistent collection of tools, prompts, and skills that an agent can use to solve a task. Given a task and a harness , an agent can attempt the task using a loop of reasoning, acting, and observing. This multi-step process generates a trajectory , which records the information read by the agent, its chain of thought, the tools used, and the final output. We denote this execution process with a prompted agent operation as . As the agent executes multiple tasks, it produces a dataset of trajectories . These trajectories often contain instances of failure and useful insights that can be used to improve the harness. Consequently, we ask whether the agent can retrospectively analyze past trajectories to optimize its harness and improve its future performance. To quantify this, we define a latent utility function that measures the quality of a trajectory. We formalize the optimization as a function that returns a modified harness . The goal is to find an optimal harness that maximizes the expected utility on future tasks:
Problem. However, estimating this utility function accurately is difficult in practice. To evaluate the true utility of a harness, we would need a representative validation set of future tasks and a mechanism to calculate the success rate of the agent using this specific harness. In our setting, the function is latent and cannot be directly observed.
Our Approach. Because the utility is latent, we cannot directly optimize it. Instead, we substitute this latent utility with a self-preference estimator. Specifically, we instruct the agent to compare multiple trajectories on the same task to compute a self-preference ranking. We define a ranking function as . This function yields a preference ordering over the given trajectories and provides a rationale that explains why the agent prefers certain executions over others. The next section details how we organize the operations of solving, ranking, and optimizing to improve the latent harness utility.
4 Retrospective Harness Optimization
We propose RHO, a self-supervised method that improves a harness using only past trajectories. Specifically, our pipeline (Figure 2) consists of three stages, namely coreset selection, group rollout, and best-of- harness proposal. First, we select a representative subset of past tasks to define the optimization target. Next, we sample a group of parallel rollouts for each task in this coreset and extract harness improvement signals from them. Finally, we generate candidate harnesses based on these signals and retain the most preferred one using pairwise self-preference. The full algorithm is detailed in Algorithm 1.
4.1 Coreset Selection
Given a large set of past trajectories, we need to extract the most critical signals to guide the harness optimization. Optimizing the harness on every individual trajectory is computationally prohibitive, and it further risks diluting important signals with trivial ones. To address this, we first select a coreset from the full set to represent the trajectories that require optimization the most. Specifically, we require the coreset to capture both challenging and diverse scenarios. This requirement encourages our optimization to cover a wide range of failure modes when addressing the most difficult problems. To accomplish this, we introduce a Determinantal Point Process (DPP) kernel (Kulesza and Taskar, 2012) to rank all past trajectories by difficulty while satisfying a diversity constraint. In practice, we employ a language model judge to analyze every trajectory and extract a difficulty score alongside a textual description. This description details the specific challenges of the problem and potential failure modes. We then compute the embedding of this description and use the cosine similarity between embeddings as the similarity metric for any two trajectories and . By considering both the difficulty scores and the trajectory similarity matrix , we construct a kernel matrix
where is a scaled version of the trajectory’s difficulty score :
With this kernel function , DPP selects a subset with probability proportional to the kernel determinant , using parameter to adjust the relative importance of difficulty and diversity via . With , the trajectories are ranked purely by difficulty and (uniform weights) ranked purely by similarity diversity. Using , we select trajectories into a coreset that covers difficult, diverse failure modes for the subsequent stages.
4.2 Group Rollout
Inspired by previous work that uses relative advantages within a group as reward signals for reinforcement learning (Shao et al., 2024), we generate a set of trajectories by running parallel agent solves on each coreset task. Subsequently, the agent compares these group trajectories to identify underperforming runs. The agent then uses contrastive signals within the group to formulate instructions for optimizing the harness. Specifically, we perform this self-preference analysis along two dimensions.
-
Self-validation (). This dimension examines the correctness of the agent within each trajectory. The agent inspects each trajectory against the required task and environment observations to determine whether the objective is efficiently achieved, exploiting the partial ability of models to recognize the limits of their own knowledge (Pan et al., 2025). During this process, it flags incorrect tool invocations, false assumptions, and premature stopping. These flagged aspects are then extracted as areas of improvement for the relatively underperforming runs.
-
Self-consistency (). This dimension examines whether the behavior of the agent remains consistent across different trajectories. Because low self-consistency typically indicates high uncertainty (Wang et al., 2022; Farquhar et al., 2024), we instruct the agent to analyze contradictions among trajectories. The agent identifies consequential disagreements, such as divergent plans, tool sequences, or final answers, and generates optimization instructions to encourage more consistent behavior.
These and analyses yield structured evaluations in JSON format, and for each task their union forms the improvement instruction . As a result, we merge across all tasks in the coreset to form the final harness improvement instructions.
| Harness | SWE-Bench Pro | Terminal-Bench 2 | GAIA-2 | ||||
| Method | Architecture | Pass | Pass | Pass | |||
| Vanilla Codex | None | 0.59 | n/a | 0.71 | n/a | 0.29 | n/a |
| Dynamic Cheatsheet (Suzgun et al., 2025) | Skills | 0.62 | 0.73 | 0.30 | |||
| ReasoningBank (Ouyang et al., 2025) | Memory | 0.61 | 0.73 | 0.28 | |||
| Sleep-time Compute (Lin et al., 2025) | Memory | 0.64 | 0.73 | 0.32 | |||
| RHO | Skills+Tools | 0.78 | 0.76 | 0.37 | |||
4.3 Best-of- Harness Proposal
After obtaining the improvement instructions, we optimize the harness by providing these instructions to the agent. However, as observed in prior studies on agent evolution (Agrawal et al., 2025; Hu et al., 2024; Lee et al., 2026), harness optimization is inherently stochastic and may not reliably improve performance even with valid input signals. To mitigate this limitation, we sample harness proposals in parallel and filter them using agent self-preference. This selection is designed to favor candidates whose improvements generalize to future tasks. Specifically, we execute parallel optimization calls to generate candidate harnesses, denoted as to . Following this step, we use these candidates to obtain sets of new trajectories on the coreset tasks. For every coreset task, we then compute an agent preference score by ranking the new trajectory from each candidate harness against the old trajectory from the original harness. We aggregate these scores across the coreset to determine the relative advantage score of each candidate:
where is the original harness trajectory for task . Finally, we return the candidate harness with the maximum relative advantage to replace the original one. We accept this update only if its score is strictly greater than zero ().
5 Experiments and Results
Setup. We use the Codex agent (OpenAI, 2025) as the base harness for retrospective optimization. Specifically, this agent uses GPT-5.5 (OpenAI, 2026) configured with high reasoning effort. When we invoke Codex to solve a task, we construct the harness as a configurable workspace folder. This folder contains executable scripts as tools, along with text files for skills and instructions. For all our experiments, we set the coreset size to 10. In addition, we use 3 for both parallel trajectory sampling and harness proposals. To measure the improvement, we report the pass rate on the held-out test set using both the vanilla Codex harness and the optimized one.
Data. We collect past trajectories from existing benchmark datasets. Specifically, we divide the original benchmarks into a trajectory set and a test set. We then run the vanilla Codex agent on the trajectory set to generate the required trajectories for RHO. Building on this, we evaluate RHO on SWE-Bench Pro, Terminal-Bench 2, and GAIA-2. SWE-Bench Pro contains long-horizon software-engineering tasks requiring repository-level reasoning and multi-file edits (Deng et al., 2025). Terminal-Bench 2 contains command-line tasks with executable graders (Terminal-Bench Team, 2025). GAIA-2 evaluates LLM agents in dynamic, asynchronous environments for knowledge work (Froger et al., 2026). As a result, these three benchmarks cover a wide range of task types across software engineering, technical work, and knowledge work. We provide detailed information about the benchmarks and data splits in Appendix E.
5.1 Comparison with Feedback-Free Baselines
We compare RHO against three competitive harness optimization methods that do not require validation feedback. For the baselines, Dynamic Cheatsheet maintains a running record of useful facts and procedures (Suzgun et al., 2025). ReasoningBank stores reusable reasoning patterns and retrieves the top- relevant entries at inference time (Ouyang et al., 2025). Similarly, Sleep-time Compute preprocesses past traces offline into compact notes, which are then prepended to the agent’s context (Lin et al., 2025). We adapt each baseline to our datasets and agent setting while holding the total agent-call budget approximately fixed to ensure a fair comparison. Detailed adaptation procedures are provided in Appendix F.
As Table 1 shows, RHO delivers consistent improvements across all three benchmarks, whereas the baselines do not. Most notably, we achieve an absolute improvement of 19% on SWE-Bench Pro without relying on any validation-based grading. We attribute this advantage to the more flexible harness optimization that RHO enables. Specifically, the agent can create new tools, skills, and instructions for the harness, whereas previous methods focus predominantly on memory systems or text-based skills. Furthermore, the use of self-preference may contribute to the consistency of these harness improvements. In contrast, the performance gains of the baseline methods tend to be smaller and vary across different datasets. In the next section (Section 5.2), we examine how RHO modifies the harness to improve the agent.
5.2 What the Optimized Harness Contains
Figure 3 summarizes and interprets the new harness contents generated after RHO optimization. In our work, the harness is materialized as a directory containing markdown files for instructions and skills, as well as executable scripts for tools.
Across all three benchmarks, RHO adds multiple new skills and tools to the harness. Many of these new additions address typical failure modes encountered by the original harness. For example, in SWE-Bench Pro, the agent learns that the Go toolchain resides at a non-standard location outside the default path. It also discovers that Python cache directories must be stripped before producing the final diff, as failing to do so often prevents patches from applying cleanly. To address these, the agent adds a new check_build_and_lint tool that locates non-standard toolchains and flags the generated artifacts that must be kept out of the patch, fixing the diff-hygiene procedures the original trajectories repeatedly missed. These examples illustrate how RHO identifies useful tools and skills across diverse scenarios by analyzing past failures.
5.3 Comparison with Validation-Feedback Optimization
We next compare RHO against Meta-Harness (Lee et al., 2026). Meta-Harness is a validation-feedback optimizer that proposes harness edits, grades each candidate on a labeled validation split, and retains the edit that yields the highest validation pass rate. To maintain a fair comparison, we use the same Codex agent as the Meta-Harness proposer and solver. Unlike RHO, Meta-Harness requires held-out labels, and because it evolves over multiple rounds, it consumes more agent calls. Therefore, we evaluate it at a single round to match our compute budget, as well as in an extended-budget setting where it runs for ten rounds.
| Method | Val. labels | Agent calls | SWE-Bench Pro |
| RHO | none | 103 () | 0.78 |
| Meta-Harness (1 round) | required | 41 () | 0.62 |
| Meta-Harness (10 rounds) | required | 320 () | 0.80 |
At the matched single-round budget, Meta-Harness selects its best candidate using validation scores but achieves only a 0.62 pass rate on SWE-Bench Pro. This is substantially lower than the 0.78 pass rate achieved by RHO. Scaling Meta-Harness to a 10-round setting increases its performance ceiling to 0.80 on SWE-Bench Pro. However, this higher performance requires roughly three times the optimization-phase compute of RHO and, more importantly, still depends on held-out labels that RHO does not use.
6 Discussion
6.1 How does agent behavior change after optimization?
Although RHO creates new skills and tools for the agent, it is not immediately apparent through which mechanisms these updates enable the agent to perform better on future tasks. To examine this, we visualize the frequency of tool calls and the cumulative success rate with respect to the number of steps taken by the agent. Specifically, Figure 4 plots the cumulative fraction of held-out tasks resolved within a given number of agent steps, and it shows how the action mix of the agent changes over time. We observe that the performance improvements across all three datasets primarily originate from higher success rates on tasks requiring long horizons. In contrast, the gains concentrate on long-horizon tasks rather than those completed in fewer steps, most clearly on SWE-Bench Pro. Furthermore, the optimization process changes the working patterns of the agent. Consequently, the optimized agent shifts toward relying more heavily on specific types of actions. For example, on SWE-Bench Pro, the agent verifies its work much more frequently. This proactive verification appears to account for a large portion of the performance gains on long-horizon tasks. On Terminal-Bench 2 and GAIA-2, the agent increases its accuracy by actively applying newly developed tools. Complementary to our action-level statistics, token-level attribution over long reasoning chains offers a finer-grained lens on long-horizon agent behavior (Pan et al., 2026b).
6.2 How does coreset selection shape capability evolution?
Dataset Mean Chosen Std Lowest SWE-Bench Pro 0.79 0.78 0.06 0.73 Terminal-Bench 2 0.74 0.76 0.03 0.71 GAIA-2 0.34 0.37 0.03 0.32
Variant SWE Pro TB 2 GAIA-2 Full diagnosis 0.78 0.76 0.37 self-consistency 0.56 0.75 0.27 self-validation 0.70 0.73 0.30 Raw trajectory 0.60 0.75 0.29
We investigate how coreset selection influences the optimization process. To this end, we compare our DPP-based selection against several ablation strategies. These variants include selecting tasks solely by difficulty (), selecting tasks purely to maximize coverage (), and sampling trajectories randomly. In addition, we measure the final performance of the optimized harness under each selection strategy. As the t-SNE projection of task embeddings in Figure 5 reveals, selecting tasks exclusively by difficulty causes the chosen samples to cluster in a narrow region of the task distribution. This clustering occurs because the language model judges certain types of tasks as inherently more difficult, and consequently it fails to include other task types in the coreset. As a result, this strategy yields no meaningful performance improvement after optimization. Similarly, optimizing solely for coverage also produces suboptimal results. In contrast, random sampling can occasionally select a trajectory that proves useful for optimization. These findings suggest that a coreset selection strategy balancing both difficulty and diversity is vital for providing the proper signals to guide harness optimization.
6.3 Does RHO produce consistent harness updates?
Because RHO operates without ground-truth labels for reference, we evaluate whether its optimization outputs are consistent across runs. This consistency is closely tied to the overall effectiveness of our best-of- harness proposal. We examine whether the selection strategy reliably identifies the candidate harness that performs best on downstream tasks. In this experiment, we measure the test scores of all three generated candidate harnesses rather than just the most preferred one. Here Table 3 shows that the generated harnesses exhibit only moderate variance. Notably, even the lowest-scoring candidate still meaningfully improves agent performance over the baseline. Furthermore, the best-of- selection prevents the deployment of poorly performing harnesses. Specifically, the chosen harness scores higher than the worst candidate across all three benchmarks. At the same time, the most preferred harness does not invariably coincide with the highest-scoring candidate on the test set, though the selection consistently avoids the worst candidate.
6.4 How much does retrospective analysis contribute?
We analyze the contribution of the two signals extracted during retrospective analysis, namely self-validation and self-consistency. We compare these explicit signals against a more direct approach that provides raw trajectories during optimization and skips the explicit retrospective analysis step. In addition, we study the individual contribution of each diagnostic signal to the final performance.
To investigate the necessity of the group rollout and diagnostic stages, we conduct an ablation study. Specifically, we remove the self-validation and self-consistency signals independently, and we subsequently rerun the optimization and evaluation procedures. We also introduce a raw trajectory baseline. This baseline bypasses the separate ranking analysis. Instead, it provides the original trajectories directly to the optimization step, asking the agent to analyze the trajectory and propose improvements in a single pass. Table 4 shows that removing either the self-consistency or the self-validation signal consistently degrades final performance across benchmarks. This result indicates that both signals are highly important for optimizing the harness. Additionally, full diagnosis outperforms the simplified raw-trajectory baseline on all three benchmarks. This result suggests that although single-pass trajectory analysis is feasible, the explicit self-validation and self-consistency signals are essential rather than incidental, yielding more reliable improvements across datasets.
7 Conclusion
We introduced RHO, which reframes harness improvement as a retrospective process that an agent can run on its own past experience, rather than a search guided by external ground-truth feedback. The central idea is that an agent’s own trajectories already contain the signal needed to improve it, since re-solving past tasks and comparing the outcomes exposes where the harness fails and what would fix it. Across all three domains (software engineering, technical work, and knowledge work), this self-supervised loop yields consistent held-out gains and reshapes how the agent works. We see RHO as a step toward agents that keep improving from the experience they accumulate in deployment, where labeled validation data is rare.
Limitations
In this paper we introduce RHO, a self-supervised method that improves an agent’s harness from its own past trajectories without any external grading. However, operating without ground-truth feedback carries several limitations. First, group rollout replays each coreset task several times, which assumes environments that reset cleanly and tolerate repeated attempts, leaving one-shot or irreversible tasks outside the setting RHO targets. Second, RHO presumes that a meaningful portion of the agent’s competence is mediated by an editable harness of prompts, skills, and tools; our experiments span software engineering, technical work, and knowledge work, and extending RHO to domains with different harness surfaces, task fingerprints, and rollout budgets remains future work. Third, RHO trusts past trajectories as its only optimization input; in open environments these trajectories can embed adversarial content injected mid-task (Liu et al., 2026), and harness updates distilled from compromised trajectories could entrench such behavior, reinforcing the need for the audit practices in the Ethics Statement.
Ethics Statement
RHO modifies persistent agent behavior from model-generated judgments. This can amplify mistaken preferences, unsafe procedures, or biased behavioral rules if the evaluator prefers them. Deployments should keep full audit logs, require human approval for sensitive harness edits, and use domain-specific safety checks before applying accepted harnesses to high-impact tasks.
Reproducibility Statement
Every run persists prompts, completions, trajectories, diagnoses, candidate harnesses, harness diffs, configs, scores, run metadata, and held-out reports. The numbers in this draft are direct reads from recorded run reports. A detailed comparison with related work is given in Appendix A, prompts are listed in Appendix B, hyperparameters in Appendix C, full pipeline details in Appendix D, per-dataset specifications in Appendix E, baseline implementations in Appendix F, and per-method optimization-phase compute cost in Appendix G.
References
- Agrawal et al. (2025) Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2025. GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457.
- Deng et al. (2025) Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, and 3 others. 2025. SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941.
- Farquhar et al. (2024) Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625–630.
- Froger et al. (2026) Romain Froger and 1 others. 2026. Gaia2: Benchmarking LLM agents on dynamic and asynchronous environments. In International Conference on Learning Representations (ICLR). ArXiv:2602.11964.
- Hu et al. (2024) Shengran Hu, Cong Lu, and Jeff Clune. 2024. Automated design of agentic systems. arXiv preprint arXiv:2408.08435.
- Khattab et al. (2023) Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714.
- Kulesza and Taskar (2012) Alex Kulesza and Ben Taskar. 2012. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 5(2–3):123–286.
- Lee et al. (2026) Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. 2026. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052.
- Lin et al. (2025) Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, and Joseph E. Gonzalez. 2025. Sleep-time compute: Beyond inference scaling at test-time. arXiv preprint arXiv:2504.13171.
- Lin et al. (2026) Minhua Lin, Zhiwei Zhang, Hanqing Lu, Hui Liu, Xianfeng Tang, Qi He, Xiang Zhang, and Suhang Wang. 2026. MemMA: Coordinating the memory cycle through multi-agent reasoning and in-situ self-evolution. arXiv preprint arXiv:2603.18718.
- Liu et al. (2026) Zhichao Liu, Wenbo Pan, Haining Yu, Ge Gao, Tianqing Zhu, and Xiaohua Jia. 2026. WebTrap: Stealthy mid-task hijacking of browser agents during navigation. arXiv preprint arXiv:2605.08310.
- OpenAI (2025) OpenAI. 2025. OpenAI Codex. https://developers.openai.com/codex/.
- OpenAI (2026) OpenAI. 2026. Introducing GPT-5.5. https://openai.com/index/introducing-gpt-5-5/.
- Ouyang et al. (2026) Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, Maohao Shen, Vishy Tirumalashetty, George Lee, Jiawei Han, Tomas Pfister, and Chen-Yu Lee. 2026. SkillOS: Learning skill curation for self-evolving agents. arXiv preprint arXiv:2605.06614.
- Ouyang et al. (2025) Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. 2025. ReasoningBank: Scaling agent self-evolving with reasoning memory. arXiv preprint arXiv:2509.25140.
- Packer et al. (2023) Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560.
- Pan et al. (2026a) Wenbo Pan, Shujie Liu, Xiangyang Zhou, Shiwei Zhang, Wanlu Shi, Mirror Xu, and Xiaohua Jia. 2026a. M⋆: Every task deserves its own memory harness. arXiv preprint arXiv:2604.11811.
- Pan et al. (2026b) Wenbo Pan, Zhichao Liu, Xianlong Wang, Haining Yu, and Xiaohua Jia. 2026b. Towards long-horizon interpretability: Efficient and faithful multi-token attribution for reasoning LLMs. arXiv preprint arXiv:2602.01914.
- Pan et al. (2025) Wenbo Pan, Jie Xu, Qiguang Chen, Junhao Dong, Libo Qin, Xinfeng Li, Haining Yu, and Xiaohua Jia. 2025. Can LLMs refuse questions they do not know? Measuring knowledge-aware refusal in factual tasks. arXiv preprint arXiv:2510.01782.
- Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
- Suzgun et al. (2025) Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. 2025. Dynamic cheatsheet: Test-time learning with adaptive memory. arXiv preprint arXiv:2504.07952.
- Terminal-Bench Team (2025) Terminal-Bench Team. 2025. Terminal-Bench 2: Benchmarking agents on hard, realistic command-line tasks. https://www.tbench.ai/. Citation details TBD.
- Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
- Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2023. C-Pack: Packed resources for general chinese embeddings. arXiv preprint arXiv:2309.07597.
- Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
- Yuksekgonul et al. (2024) Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. TextGrad: Automatic “differentiation” via text. arXiv preprint arXiv:2406.07496.
- Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910.
Appendix A Comparison with Related Work
| Satisfies RHO’s setting | ||||||
| Method | Harness architecture | Feedback signal | Cost | Label- free | Full harness | Single pass |
| Validation-feedback optimization | ||||||
| OPRO (Yang et al., 2023) | Prompt | Validation metric | Iterative search | ○ | ○ | ○ |
| DSPy (Khattab et al., 2023) | Prompt demos | Validation metric | Iterative search | ○ | ○ | ○ |
| TextGrad (Yuksekgonul et al., 2024) | Prompt | Textual gradient | Iterative search | ○ | ○ | ○ |
| GEPA (Agrawal et al., 2025) | Prompt | Val. metric reflection | Iterative (genetic) | ○ | ○ | ○ |
| ADAS (Hu et al., 2024) | Full agent code | Validation accuracy | 25–30 iter. | ○ | ● | ○ |
| Meta-Harness (Lee et al., 2026) | Full harness code | Search-set score | 20 iter. | ○ | ● | ○ |
| Experience-based self-improvement | ||||||
| Dynamic Cheatsheet (Suzgun et al., 2025) | Cheatsheet text | Self-judgment | Online stream | ● | ◐ | ○ |
| ReasoningBank (Ouyang et al., 2025) | Memory items | LLM-as-judge | Online stream | ● | ○ | ○ |
| MemMA (Lin et al., 2026) | Memory entries | Synthetic probe QA | Online (session) | ● | ○ | ○ |
| Sleep-time Compute (Lin et al., 2025) | Input context | None | Precompute | ● | ○ | ● |
| SkillOS† (Ouyang et al., 2026) | Skill list | RL reward† | RL training | ◐ | ◐ | ○ |
| RHO (ours) | Tools skills instr. | Self-preference | Single pass | ● | ● | ● |
†SkillOS’s reinforcement-learning reward is an LLM-as-judge self-signal rather than gold labels, but it still requires a labeled training corpus, a separate judge model, and weight updates.
Table 5 compares RHO with the prior methods discussed in Section 2. For each method we report the harness surface it edits, the feedback signal that drives it, and the cost regime it incurs, and we mark whether it meets RHO’s setting on three criteria, namely whether optimization is label-free, whether it edits the full harness, and whether it runs as a single offline retrospective pass. Validation-feedback optimizers reach the full-harness end of the surface axis, where ADAS and Meta-Harness rewrite executable code, but every member of this family steers the search with a ground-truth metric and iterates against it. Experience-based methods drop the labels, yet they only curate memory or skill text online and leave the executable harness untouched, while Sleep-time Compute precomputes context offline but optimizes nothing. As a result, each prior method satisfies at most two of the three axes, and RHO is the only entry that satisfies all three at once.
Appendix B Prompts
This appendix collects the prompts that instantiate the five agent operators of RHO (§3, Algorithm 1, Figure 2), namely , the difficulty judge used in Coreset Selection, the diagnosis analysis, , and . The prompts are reproduced verbatim. Placeholders of the form {name} are filled at call time with the values described under each block.
B.1 Solve
Every call materializes the harness and the task into a fresh workspace at harness/ and task/ and hands the agent the wrapper instructions below. The wrapper is task-agnostic, in that the harness directory carries all task-shaping guidance, and the agent is told only how to read the workspace and how to deliver a final answer. The same wrapper is reused for the baseline rollout and for every candidate-harness rollout so that is the only varying input.
B.2 Coreset Selection (Difficulty Judge)
The difficulty judge produces the score and the abstract fingerprint that drive the DPP coreset selector of §4.1. It sees the task description together with a length-bounded digest of one short prior trajectory under the current harness. The digest is truncated head/tail to a fixed token budget, and any commands that read the task’s expected-answer files are scrubbed before the digest is shown to the judge. The judge is asked to keep the fingerprint in task-agnostic structural vocabulary so that fingerprints from different codebases remain comparable under cosine similarity.
{query}
is the natural-language task description.
{trajectory_digest}
is the scrubbed, head/tail-truncated digest of one prior trajectory for the same task. This difficulty value is the score that enters the DPP kernel, where it is normalized by as in §4.1, and the fingerprint is embedded to a unit vector that defines the similarity matrix .
B.3 Diagnosis
The diagnosis prompt implements (§4.2). The workspace presents the agent with the original task, the shared harness used by every rollout, and rollout directories. The agent executes a five-step workflow, comprising per-trajectory inspection, failure-mode analysis (self-validation), cross-trajectory disagreement analysis (self-consistency), a single high-level harness improvement direction, and a severity score that doubles as a soft attention weight downstream. The structured JSON output binds each field to a fixed slot so can attend by severity.
The trajectory directories carry the full event stream, the agent’s final message, and any workspace diff produced by the rollout. The
self-validation
self-consistency
ablations in Table
remove the corresponding step and the corresponding output field, and the surrounding scaffolding is left intact.
B.4 Optimization
The optimization prompt implements (§4.3). The editor agent is given write access to a fresh copy of the harness and a directory of per-task diagnoses sorted by severity. The instruction frames severity as a soft attention weight, requires cross-task pattern matching before any edit, and discourages task-specific hardcoded fixes. Each of the candidates is sampled independently with the same prompt, and randomness from the underlying agent supplies the diversity.
Each
diagnoses/task_XXXX/
subdirectory holds the serialized diagnosis JSON rendered as Markdown alongside the original task prompt. Subdirectories are indexed in descending severity so the agent encounters the most consequential diagnoses first.
B.5 Pairwise Ranking
The ranking prompt implements used in Best-of- acceptance (§4.3 and Algorithm 1). The evaluator sees the task, the two harnesses, and the two trajectories. The candidate trajectory is presented as trajectory_A and the baseline as trajectory_B, and the orchestrator negates the returned integer so the scalar score is oriented as baseline candidate regardless of presentation order. Presenting the candidate first reduces a later-option preference bias we observed in pilot runs. The rubric below uses an integer scale in , and downstream only the sign and relative magnitude of the score are used (§3).
The candidate score in Algorithm
averages this integer (with the orientation flip described above) over the coreset , and the candidate is accepted only when , otherwise the harness remains at .
Appendix C Hyperparameters and Infrastructure
Table 6 lists every hyperparameter and infrastructure setting used in our experiments. Values already reported in the main text (Section 3 and Section 5.1) are reproduced here for convenience, and values not previously stated are introduced here. The intent is an audit-ready specification, with one column per parameter, one value per cell, and no defaults left implicit. Dataset-specific overrides on solver wall-clock and grading are deferred to Appendix E, and prompt text for every agent role is in Appendix B.
| Parameter | Value | Description |
| Backbone agent | ||
| Model† | Codex gpt-5.5 | Shared across solve, optimize, and rank. |
| Reasoning effort† | high | Applied to all roles uniformly. |
| Sampling temperature | provider default | Codex reasoning models do not expose temperature. |
| Provider | Cloud-hosted gpt-5.5 (Codex CLI) | Single hosted endpoint. |
| Coreset selection | ||
| Selector | DPP, greedy MAP | Kulesza and Taskar (2012). |
| Coreset size † | 10 | Train tasks. |
| DPP weight † | 0.7 | Difficulty/diversity tradeoff. |
| Score floor | 0.1 | Lower bound on normalized difficulty. |
| Judge model | Codex gpt-5.5, high | Same backbone as the solver. |
| Trajectory-digest budget | 10,000 BPE tokens, head/tail truncation | Ground-truth-revealing commands scrubbed. |
| Fingerprint embedding | BAAI/bge-large-en-v1.5, 1024-d | Executed locally (Xiao et al., 2023). |
| Group rollout | ||
| Rollouts per task | 3 | Parallel solves under per coreset task. |
| Solve wall-clock timeout | 900 s | Dataset overrides in Appendix E. |
| Diagnosis prompt | see Appendix B | Self-validation self-consistency cues. |
| Severity range | Soft attention weight, not a hard threshold. | |
| Best-of- harness proposal | ||
| Candidates † | 3 | Parallel optimizer samples. |
| Optimizer prompt | see Appendix B | Conditioned on . |
| Ranking timeout | 300 s per pairwise call | Per . |
| Acceptance threshold | (strict) | Mean pairwise score over the coreset. |
| Infrastructure | ||
| Agent-call concurrency | 10 concurrent calls (cap 30) | Parallelism for Table 1 runs. |
| Rounds per experiment | 1 | Unless stated otherwise. |
| Persistence | full logs to disk | Prompts, completions, trajectories, diagnoses, |
| candidate harnesses, diffs, scores, run summaries. | ||
Several choices in Table 6 warrant brief justification. We set because three rollouts are enough for the diagnosis prompt to surface cross-trajectory disagreements, and pilots with larger inflated cost linearly without sharpening the diagnosis signal. We weight the DPP at so that difficulty dominates diversity, on the principle that an unselected easy task carries less optimization signal than a redundant hard one. Concretely, enters through the difficulty weights with , and the factor of two offsets the difficulty term’s doubled appearance in , so that and weight the difficulty and diversity terms directly. We make the acceptance gate strictly positive rather than non-negative because pairwise self-preference is a noisy estimator, and breaking ties in favor of change would inflate regression risk for no expected gain. Finally, we use the same Codex gpt-5.5 backbone for solver, optimizer, and ranker, since decoupling the solver from the judge would introduce a confound where measured held-out gains could be attributed to a stronger judge rather than to RHO itself. Per-dataset configuration overrides referenced above are in Appendix E, and baseline-specific configurations are in Appendix F.
Appendix D Pipeline Implementation Details
This appendix documents the design choices that govern RHO’s behavior but do not appear in §4 or Algorithm 1. The equations and operator signatures are fixed by the main text, and what follows are the mechanics that make them executable.
D.1 Harness Representation and Mounting
A harness is a directory of files with no fixed schema, where prose instructions, executable scripts, and structured configuration sit side by side. At every operator call the harness is materialized as a subdirectory of the agent’s working directory, and the agent reads it the same way it reads task files rather than as a system-prompt prefix. The operator mounts the harness read-only by convention, while mounts a fresh copy with write access and accepts whatever the agent leaves behind on the filesystem. Harnesses are compared by content, so if returns a harness identical to the input, the candidate is treated as a no-op and dropped before evaluation. This keeps the harness surface tool-agnostic and lets the optimizer extend any of the three file kinds without a representation change.
D.2 Role Separation and Workspace Isolation
The same backbone executes , the diagnosis analysis (/), , and . Role separation is achieved by workspace contents rather than by changing the backbone, where each operator runs in a fresh workspace that contains only the inputs it should see. sees the task files and the current harness. The diagnosis analysis sees the task and the group of trajectories for that task. sees the harness directory, the diagnosis instructions, and the trajectory rollouts that motivated them. Finally, sees the task, two trajectories, and both candidate harness directories side by side. Per-role ablation therefore amounts to swapping prompts and inputs, not models, which keeps the operators directly comparable.
D.3 Order, Parsing, and Failure Handling in Pairwise Ranking
is invoked once per (task, candidate) pair. We present the candidate trajectory first and the baseline trajectory second, then negate the parsed scalar. This swap is a standard mitigation for position bias in pairwise judges, applied without per-call order randomization. The judge is constrained to return a single integer in with a one-sentence rationale, and we do not retry. Any parse or execution failure deterministically yields zero. This makes a strict pessimist, since silent failures pull the mean toward the rejection threshold rather than away from it.
D.4 Diagnosis vs. Ranking Inputs
The diagnosis analysis consumes the full group of rollouts for one task and produces a single textual instruction with a severity weight, whereas is strictly pairwise. We do not collapse a group into a Bradley–Terry score for evaluation. For each candidate, we pair the candidate’s rollout of a task against a fixed baseline trajectory drawn from the original group, and we hold this baseline constant across every candidate so the comparison stays anchored to the same reference. The consequence is that enters the pipeline as a diagnostic device only, since it sharpens the instruction through cross-trajectory inconsistency but does not vote on which candidate wins.
D.5 Optimizer Action Space
is a code-agent invocation, not a constrained text rewriter. It sees the materialized harness as a filesystem and may add, remove, or modify any file inside it. We hand it the diagnosis instructions sorted by descending severity, with severity exposed as a soft attention weight in rather than a hard priority queue, so the agent can still merge or skip suggestions that conflict. We do not parse a diff out of the agent’s final message, and the new harness state is recovered from the directory itself when the call returns. This keeps the action space identical to the representation, where anything that fits in a directory is a valid edit.
D.6 Acceptance Gate and No-Ops
An update is accepted only when the best candidate’s mean pairwise score over the coreset is strictly positive, and a mean of zero is rejected. Before the gate, a candidate is dropped if the optimizer failed, the call timed out, or the produced harness is identical to the input. If every candidate is dropped, or every surviving candidate scores , RHO leaves the harness unchanged. This no-update behavior is part of the result rather than a tooling failure. In Figure 5, selectors whose coreset fails to expose useful weaknesses (pure coverage, pure difficulty) leave held-out accuracy near the Vanilla Codex baseline.
D.7 Persistence
We persist the input harness id, along with the trajectories produced by each operator (, the diagnosis analysis, , and ) identified by id. We further persist the diagnosis instruction, the candidate harness ids together with their diffs against the input, the per-candidate pairwise scores, the mean pairwise score, and the accept flag. Every stored trajectory carries its full event stream, final message, workspace diff, and wall-clock time. These records are what makes downstream audit, re-grading, and ablation possible without re-running the agent.
Appendix E Dataset Specifications
This appendix documents what we take from each upstream benchmark, how we split it, what the agent sees at solve time, and how grading runs. Pinned upstream commits make the partition and grader reproducible from a clean checkout.
E.1 SWE-Bench Pro
SWE-Bench Pro (Deng et al., 2025) is the long-horizon software-engineering benchmark in our suite. Tasks resolve only when a multi-file patch passes the upstream test set, and failure modes are therefore concrete and traceable to the harness’s repository conventions and build commands.
Source. We load the test split of ScaleAI/SWE-bench_Pro from Hugging Face. Evaluator scripts and per-instance Docker images come from the official scaleapi/SWE-bench_Pro-os repository, pinned to a fixed commit.111Commit 0c64e26f00b9c190432de7fc520c8ceed5c25518.
Split. Rows are ordered by the SHA-256 hash of (seed, instance_id) with the seed fixed. The first 100 form the training pool and the next 100 the held-out test pool, and the remaining rows are unused. We report on the held-out test pool.
Solve interface. Each task materializes a prompt file describing the issue plus a fresh clone of the upstream repository at its base commit, mounted in the workspace. The agent edits repository files in place, and no extra tools are injected.
Grading. We extract the agent’s patch by re-applying its workspace edits to a fresh checkout and taking git diff --binary. If the workspace path fails, we fall back to scanning the agent’s final message for fenced diff blocks. Binary hunks and auto-generated paths (dependency, cache, and build directories) are stripped before scoring. The patch is then applied inside the official per-instance Docker image distributed by the SWE-Bench Pro authors, and the official run and parser scripts are invoked. A task passes iff every and test resolves correctly. The Docker wall-clock budget is one hour per task.
E.2 Terminal-Bench 2
Terminal-Bench 2 (Terminal-Bench Team, 2025) contains executable command-line tasks. Failure is a missed reward, not a missed convention, so the harness mostly buys consistency in how the agent inspects state and chains commands.
Source. We use the upstream Terminal-Bench 2 repository at a fixed commit222Commit 53ff2b87d621bdb97b455671f2bd9728b7d86c11., yielding 89 tasks.
Split. Tasks are hash-ordered with the same seeded scheme as SWE-Bench Pro. The first 30 form the training pool, the remaining 59 the held-out pool.
Solve interface. Each task spins up a fresh Docker container whose image is declared by the task. Containers carry a cleanup label and per-task CPU and memory limits. The agent’s host workspace is bind-mounted inside the container. The solve prompt instructs the agent to author shell scripts on the host and execute them inside the container. A wall-clock watchdog terminates the container when the task’s declared agent timeout elapses.
Grading. We run the task’s upstream test suite inside the still-running container under its declared verifier timeout. The verifier writes a reward of 0 or 1 to a known path. A task passes iff the reward equals 1. We apply no difficulty filter, so held-out numbers span the upstream difficulty mix.
Data integrity. The solve prompt instructs the agent not to read the test directory, the upstream solution, or the verifier log. This is a prompt-level convention, not a sandbox restriction. We document it for transparency, since a sufficiently adversarial agent could read the verifier and game the reward. We rely on the held-out pool and Table 1’s cross-benchmark consistency to detect such behavior, and we have not observed it.
E.3 GAIA-2
GAIA-2 (Froger et al., 2026) differs from the two coding benchmarks in that the environment evolves independently of the agent. The harness layer therefore has to encode behavior under partial observation and asynchronous events, not just stable build conventions.
Source. We load the validation split of the GAIA-2 release on Hugging Face under the mini configuration, yielding 200 scenarios. Each scenario ships an asynchronous event stream and an upstream write-action verifier.
Split. Scenarios are hash-ordered with the same seeded scheme. The first 100 form the training pool, and the remainder, 100, form the held-out pool. We report on the held-out slice.
Solve interface. Each scenario’s workspace contains a task prompt, a tool dispatcher, and a tool catalog. The agent invokes tools through a single shell command per call. The dispatcher relays each call to a sidecar process running the upstream environment, which advances simulated time and replays scheduled events. The scenario’s ground-truth state is held entirely by the sidecar, and the agent never reads it.
Grading. When the agent finishes, the sidecar invokes the scenario’s upstream verifier. The judge LLM is the same Codex gpt-5.5 backbone we use elsewhere, and routing defaults to the same Azure Foundry endpoint. A scenario passes iff the upstream verifier reports success.
Environment modifications. Two changes to the upstream environment affect reported numbers and must be disclosed. First, the upstream cap of one send_message_to_user call per turn is raised to four, since the original cap penalizes verbose agents that would otherwise complete the task correctly. Second, three optional judge-relaxation switches (event filtering, relaxed per-app UI judging, and trivial filesystem-read filtering) are available as ablation axes but are all disabled in the experiments reported in Table 1.
Appendix F Baseline Implementations
Every baseline runs under the same Codex gpt-5.5 backbone at high reasoning effort. The trajectory-only family (Dynamic Cheatsheet, ReasoningBank, and Sleep-time Compute) shares RHO’s Coreset Selection budget of 10 training tasks and 3 candidates. Baselines differ only in what the offline phase persists and how the solver consumes it. This appendix documents each.
F.1 Dynamic Cheatsheet (Suzgun et al., 2025)
Dynamic Cheatsheet curates a running list of reusable facts and procedures harvested from past trajectories.
Persistence layer. A single markdown file lives inside the harness, structured as <memory_item> blocks containing a description, a worked example, and a usage count.
Offline phase. A curator agent iterates over the selected training tasks in order. For each task it reads the current cheatsheet plus the solve transcript and rewrites the cheatsheet in place.
Online consumption. The cheatsheet is part of the harness, and the solver reads it as part of normal harness lookup. No additional prompt prefix is injected.
Faithfulness. Upstream Dynamic Cheatsheet updates the cheatsheet after every task, so the next task’s solver sees the update. Our default configuration runs solves for all selected training tasks before invoking the curator, then commits one cheatsheet. The single-task configuration that matches the upstream per-example update is available as a sensitivity setting, while Table 1 uses the default.
F.2 ReasoningBank (Ouyang et al., 2025)
ReasoningBank stores reusable reasoning patterns extracted from past trajectories and retrieves them on demand at inference time.
Persistence layer. A JSONL bank of items, each with title, description, body, and a precomputed embedding. The bank lives outside the harness directory because solve-time access is by similarity retrieval, not by harness materialization.
Offline phase. For each selected training task we solve it, ask the judge whether the trajectory was successful, then run an extraction prompt that distills reusable reasoning patterns from the trajectory. Extracted items are appended to the bank, and embeddings are computed locally.
Online consumption. At each held-out solve, we encode the task description with the same encoder, retrieve the top- items by cosine similarity, render them as a memory preamble prepended to the solve instructions. Top- is fixed at 1.
Faithfulness. We replace the upstream Gemini embedding with BAAI/bge-large-en-v1.5 (Xiao et al., 2023), a 1024-dimensional locally executed sentence encoder, so that every baseline that uses retrieval shares the same encoder, dimensionality, and storage backend. This removes a confound between memory-bank quality and embedding-provider quality.
F.3 Sleep-time Compute (Lin et al., 2025)
Sleep-time Compute preprocesses past traces into compact notes that are prepended to the agent’s context at inference time, following the Letta implementation (Packer et al., 2023).
Persistence layer. A set of bounded markdown memory blocks stored inside the harness, edited with structured edit tools (insert, replace, rethink, finish) that mirror upstream Letta semantics.
Offline phase. A sleep-time agent iterates over the selected training tasks. For each task it reads the task’s prompt and trajectories, then issues a sequence of memory-edit tool calls to update the harness’s memory blocks. The system prompt is the upstream Letta system prompt, used verbatim.
Online consumption. The solver reads the memory blocks as part of the harness, the same way it reads any harness file.
Scope. Edits accumulate across tasks, and the sleep-time agent works against a single evolving memory state task after task. We do not reset memory between training tasks.
F.4 Meta-Harness (Lee et al., 2026)
Meta-Harness is the validation-feedback reference, where a meta-agent rewrites the harness, and the rewrites are scored against a labeled validation set.
Persistence layer. A sequence of candidate harness directories paired with a search history recording per-candidate validation pass rates.
Offline phase. An outer loop alternates between a proposer that reads the search history and emits a new harness, and an evaluator that grades the proposed harness on the search-task set using the dataset’s ground-truth grader. Upstream’s default budget is 20 outer iterations 3 candidates per iteration 2 solve trials per task, far above the budget we give RHO.
Validation-feedback footprint. The proposer reads each candidate’s per-task scores, mean score, and pass rate from the history, and the grader signal directly shapes the next proposal. This is the hallmark of the validation-feedback family and the axis along which Meta-Harness is incomparable to the trajectory-only baselines.
Matched-budget configuration. For Table 2 we run the outer loop for one iteration with 3 candidates and 1 solve trial per task, so the candidate count matches RHO’s . The validation-grade cost is what Table 2 reports.
Shared configuration. All four baselines and RHO start from the same empty harness for a given dataset, use the same coreset of selected training tasks, and are graded against the same held-out split with the same grader. The only deliberate axis of variation is what the offline phase persists and how the solver consumes it. Meta-Harness additionally consumes validation grades, and this is the axis Table 2 measures.
| Method | rollout | diagnose | optimize | after | rank | test | total | llm |
| Vanilla Codex | 0 | 0 | 0 | 0 | 0 | 100 | 100 | 0 |
| ReasoningBank (Ouyang et al., 2025) | 10 | 0 | 0 | 0 | 0 | 100 | 110 | 20 |
| Dynamic Cheatsheet (Suzgun et al., 2025) | 30 | 0 | 10 | 10 | 10 | 100 | 160 | 0 |
| Sleep-time Compute (Lin et al., 2025) | 30 | 0 | 30 | 30 | 30 | 100 | 220 | 0 |
| RHO | 30 | 10 | 3 | 30 | 30 | 100 | 203 | 0 |
Appendix G Optimization-Phase Compute Cost
This appendix documents the optimization-phase compute cost of RHO and the baselines reported in Table 1 and Table 2. We report the numbers without claiming an efficiency advantage. Among the trajectory-only methods at matched coreset budget (, ), RHO sits in the same order of magnitude as Sleep-time Compute and Dynamic Cheatsheet, and is more expensive than ReasoningBank. The matched-budget Meta-Harness configuration is the only baseline whose cost is qualitatively different, and its upstream-default budget would be larger still, as we discuss at the end.
G.1 Accounting Scope
Every method dispatches two kinds of model calls during the offline phase, plus the same per-task solve at evaluation time. We count them separately because they go through different backends and have different unit costs.
Codex (agent) invocations. One Codex CLI subprocess that runs to completion is one agent invocation. Each invocation persists a trajectory directory recording the role (one of solve, diagnose, optimize, rank), the harness id, the task id, the wall-clock duration, and the exit status. We obtain the per-method totals by counting these directories.
Auxiliary LLM calls. A single direct API call to the same gpt-5.5 backbone, issued outside the Codex CLI. Only ReasoningBank uses these in the offline phase, namely two per training task, for success judgment and memory-item extraction. Coreset selection also issues one auxiliary LLM call per candidate task before any method starts. This cost is shared across methods at matched coreset and is reported once at the end of this appendix.
Embedding calls. ReasoningBank and DPP coreset selection both call BAAI/bge-large-en-v1.5 (Xiao et al., 2023) for fingerprint or query embedding. We execute the encoder locally, so these calls do not hit a remote endpoint and are not aggregated with the LLM totals.
G.2 Per-Method Decomposition
Table 7 breaks down the optimization phase on SWE-Bench Pro ( train tasks, held-out tasks) into agent invocations by role. Terminal-Bench 2 differs only in , while GAIA-2 matches SWE-Bench Pro.
The table is a literal accounting of how each method spends its offline budget. ReasoningBank issues one solve per training task and never proposes a candidate harness, so the offline phase costs only extra Codex invocations on top of the held-out evaluation, and its auxiliary LLM calls are the success judge and the memory-item extractor. Dynamic Cheatsheet and Sleep-time Compute both roll out each training task times to surface variance, then run an optimizer agent. Dynamic Cheatsheet invokes a single curator over all tasks ( candidate), while Sleep-time Compute runs one edit pass per task per restart (3 restarts tasks optimize calls). Both then commit each unique candidate harness through the after-solve and pairwise-rank steps shared with RHO. RHO’s footprint differs from Sleep-time Compute’s only by trading per-task optimize calls for diagnose calls plus the candidate-level optimize calls in the best-of- proposer, and the after-solve and rank phases are identical at candidates.
G.3 Wall-Clock Time
Table 8 reports two complementary wall-clock measurements per (method, dataset) cell. is the sum of per-invocation wall-clock over every agent invocation in the run, and it is the cost of running the same workload serially on one agent process. End-to-end is the elapsed time from the run’s start to its completion, under our infrastructure setting of 10 concurrent agent calls and 10 concurrent graders. The ratio between the two columns reflects how much of the workload was successfully parallelized, where the trajectory-only methods reach – speedup, while ReasoningBank reaches – because its offline phase is sequential by construction (each training task’s solve must commit to memory before the next task’s retrieval). Meta-Harness is a validation-feedback rather than a trajectory-only method, so we report its budget separately in §G.4, not here.
| SWE-Bench Pro | Terminal-Bench 2 | GAIA-2 | ||||
| Method | end-to-end | end-to-end | end-to-end | |||
| Vanilla Codex | 9.4 | 1.1 | 4.3 | 0.6 | 4.2 | 0.6 |
| ReasoningBank (Ouyang et al., 2025) | 17.5 | 4.6 | 8.0 | 2.4 | 7.2 | 2.0 |
| Dynamic Cheatsheet (Suzgun et al., 2025) | 17.9 | 2.5 | 10.8 | 1.9 | 6.5 | 1.2 |
| Sleep-time Compute (Lin et al., 2025) | 22.2 | 3.2 | 14.5 | 2.2 | 7.0 | 1.2 |
| RHO | 23.1 | 3.2 | 15.5 | 2.5 | 9.2 | 1.4 |
Two observations are worth stating explicitly. First, per-invocation cost varies by more than across datasets (GAIA-2 averages around s per invocation, Terminal-Bench 2 around s, SWE-Bench Pro around s), so an agent-call count is informative only within a dataset, and cost cannot be compared across datasets by summing invocation counts alone. Second, at matched coreset and , RHO is within of Sleep-time Compute on every dataset, more expensive than Dynamic Cheatsheet (which uses ), and substantially more expensive than ReasoningBank (which performs no group rollout, no candidate proposal, no pairwise rank).
G.4 Matched-Budget Meta-Harness
Meta-Harness has a qualitatively larger compute footprint because validation grades feed back into the proposer. At its upstream default ( outer iterations, candidates per iteration, solve trials per task) the offline phase alone would cost agent invocations on SWE-Bench Pro — nearly twelve times RHO’s optimization-phase count of . The matched-budget configuration we report in Table 2 reduces this to , , , amounting to optimization-phase invocations; this is the configuration whose held-out pass rate ( on SWE-Bench Pro) is reported alongside RHO’s in the main text. The 10-round configuration reported in the same table keeps , but raises to , giving optimization-phase invocations ( RHO’s optimization-phase count of ). These counts exclude the shared held-out evaluation, which every method pays once regardless of optimizer. Because Meta-Harness is a validation-feedback rather than a trajectory-only method, it is not included in the wall-clock comparison of Table 8.
G.5 Coreset Selection Cost (Shared)
Coreset selection runs once before any method’s offline phase begins. For SWE-Bench Pro this consumes auxiliary LLM calls (one difficulty judgment per candidate task) plus one batched embedding call over the same fingerprints, after which the selected task ids are persisted and reused verbatim by every baseline. This cost is paid once per dataset and is not attributed to any individual method.
Appendix H Optimized Harness Artifacts
This appendix reproduces, verbatim, the complete contents of the highest-scoring harness RHO produced on each benchmark, i.e. the harnesses summarized in Figure 3. Each harness is a directory of Markdown instruction and skill files together with executable tool scripts; every file is shown in full, with no abridgement. Files are grouped by benchmark and labeled with their path inside the harness directory and their role (instruction, skill, or tool).