Vatche Isahagian Vatche 关注
Gaodan Fang gaodan-fang 关注
Jayaram Radhakrishnan jayaramkr 关注
Punleuk Oum illeatmyhat 关注
Ashwath Vaithinathan Aravindan ashwath-vaithina 关注
Evelyn Duesterwald evduester 关注
G Thomas gsthomasx 关注
Vinod Muthusamy vinodmut 关注
Merve Unuvar mrvnvr 关注
在上一篇文章中，我们将 ALTK-Evolve 与 ACE 进行了对比，并展示了智能体自我蒸馏指南的交付方式——是每项任务检索几条，还是把整套指南全部注入——会同时影响准确率和成本。本文则退一步，回到更前置的问题：到底该给智能体多少记忆？为智能体配备智能体记忆听起来很简单：从它过去的工作中蒸馏经验教训，放回上下文中，经验越多就应该表现越好。但事实并非总是如此。当我们将评估扩展到八个模型——从 30B 稠密模型到前沿闭源系统——有一个发现格外突出：
智能体记忆不是一项打开即用的功能，而是一剂需要针对模型校准的剂量。
摘要
ALTK-Evolve 让智能体从自身过去的轨迹中学习：蒸馏出可复用的指南，并在推理时注入回上下文，无需更新权重，也无需人工标注。
合适的剂量因模型层级而异：有余量的强模型需要完整的指南集，较弱的模型用紧凑核心加按任务检索效果最好，而已经饱和的模型则没有可测量的提升。
精选检索可以同时是最准确也最经济的选择：gpt-oss-120b 在仅增加 +5% token 的情况下任务完成率提升了 +16.1 个百分点——而且提示词缓存让完整指南集在生产环境中也保持低成本。
核心洞察：剂量取决于能力
并非每个模型都能从相同量的记忆中获益。在横跨能力谱系的八个模型上，我们观察到了三种反复出现的模式：
有余量的强模型需要完整的指南集——包括罕见的边缘案例经验在内的每一条指南。它们有能力吸收并运用全部内容。DeepSeek-V3.2（671B MoE）在获得完整的自我挖掘指南集后，任务完成率提升了 +9.5 个百分点。
较小或较弱的模型容易被庞大的指南集淹没。对于这类模型，一个紧凑的高置信度核心指南，加上按任务检索到的少量相关指南，效果最佳。gpt-oss-120b（117B MoE）采用这种选择性方法后提升了 +16.1 个百分点——而使用完整指南集不仅提升更少，还多消耗约 50% 的 token。
已经饱和的模型没有表现出可衡量的提升。我们称之为“饱和模式”——这个标签描述的是我们观察到的现象，而非已证实的成因。该模型可能在这些任务上已经接近其能力上限，指南可能没有针对其剩余的失败点，或者模型可能没有有效地应用这些指导。在我们的运行中，GLM-5（745B MoE）就处于这种模式。
决定一个模型落入哪种模式的因素并非仅仅是参数量。基准测试的提升空间、上下文窗口大小、架构、指南质量以及任务分布似乎都会影响模型最终所处的模式，而厘清这些因素的各自作用仍在进行中。无论哪种情况，实际结论都成立：合适剂量的“记忆”取决于模型本身，而我们可以对其进行校准。
学习发生在模型周围，而非模型内部
这里的“记忆”并不意味着回放过去的对话记录。它指的是一套指南——包括行之有效的策略、需要避免的错误以及边缘情况——这些内容是从智能体自身过往的执行轨迹中提炼出来的。这个循环很直接：
智能体尝试执行任务并产生执行轨迹。
ALTK-Evolve 从其成功和失败的运行中提取行为指南。
它将这些指南整合成一套可复用的指南集。
在推理时，智能体会收到完整的指南集，或其中与任务相关的部分。
模型权重没有任何更新。这个学习循环改变的是可供智能体使用的指导，而非底层模型本身——这正是它采用成本低、并且在我们测试的八个模型中具有可移植性的原因。
全谱系结果
我们在 AppWorld 上进行了评估——该基准涵盖 9 个模拟应用（日历、消息、支付等）中的 585 个多步骤任务（168 个 testnormal + 417 个 testchallenge）。任务采用两种方式评分：智能体是否完整完成每个任务（TGC——任务目标完成率），以及场景的每个变体是否全部通过（SGC——场景目标完成率，这是一项更严格、全有或全无的指标）。完整定义见附录。
我们对比的三种配置
由于任何记忆研究的难点都在于上下文窗口中实际包含什么内容，我们事先对配置进行了定义。
两种记忆配置都来自同一套指南集，该指南集仅从 AppWorld 的训练集划分中（通过上述循环）挖掘一次。两者之间的区别仅在于这套指南的交付方式——完整指南集在每一步都注入全部内容，而精选检索则提供选定的子集——指南的生成方式从未改变，且构建过程中绝不使用任何测试集数据。
| 配置 | 智能体上下文中的内容 |
|---|---|
| 基线 | 无记忆——即智能体出厂时的原始状态。 |
| 完整指南集 | 所有挖掘出的指南，在每一步 ReAct 循环中都全部注入。 |
| 精选检索 | 上述指南中一个固定的、高置信度的核心子集，加上为每个任务检索到的少量任务相关指南（固定部分 + 可变部分）。 |
模型挖掘出的指南数量取决于其自身能力，因此我们按策略来报告配置——即“完整指南集”对比“精选检索”——而不是按原始数量，因为不同模型之间的数量不具备可比性。
三种模式，一图概览
八模型扫描中的代表性模型，以 testnormal 上的任务完成率（TGC）衡量：
图 1. 三种观察模式中的代表性模型。柱状图显示基线 vs. 最佳记忆配置在 AppWorld test\normal 上的 TGC；x 轴从 40% 开始以便差异可见。仅看 TGC 会低估更大的 SGC 提升——请参见下表中的 SGC 列。
图中绘制 TGC 以保持可读性；表格则补充了更严格的 SGC 指标，该指标上的提升往往更大：
| 模型 | 模式 | 基线 TGC / SGC | 最佳记忆 TGC / SGC | 最佳配置 | Δ TGC | Δ SGC |
|---|---|---|---|---|---|---|
| gpt-oss-120b（117B MoE） | 弱 / 选择性 | 39.9 / 21.4 | 56.0 / 37.5 | 精选检索 | +16.1 | +16.1 |
| DeepSeek-V3.2（671B MoE） | 表现强劲，仍有提升空间 | 79.8 / 64.3 | 89.3 / 80.4 | 完整指南集 | +9.5 | +16.1 |
| Claude Opus 4.6 | 表现强劲，仍有提升空间 | 90.5 / 87.5 | 94.6 / 94.6 | 完整指南集 | +4.1 | +7.1 |
| GPT-5.5 | 表现强劲（接近上限） | 92.3 / 82.1 | 95.2 / 89.3 | 完整指南集 | +2.9 | +7.2 |
| GLM-5（745B MoE） | 已饱和 | 87.5 / 80.4 | 87.5 / 80.4 | 完整指南集 | 0.0 | 0.0 |
看 SGC 这一列，更严格的指标通常比 TGC 变动更大——DeepSeek 的 SGC 提升了 +16.1 个百分点，而 TGC 只提升了 +9.5 个百分点——因为好的指南尤其能帮助智能体通过某个场景的所有变体，而不仅仅是平均情况。而且这种效果在顶尖模型上也不会消失：GPT-5.5 和 Opus 在 TGC 上都已经接近上限，但 SGC 仍然分别提升了 +7.2 和 +7.1 个百分点。只要模型还有剩余的失败模式可以针对，记忆就会持续带来回报。
最省钱的记忆策略也可能是最好的策略
一个实际顾虑：注入完整的指南集会膨胀每个 ReAct 步骤的输入，因为指南会在每一轮重新发送。以下是我们观察到的结果：
| 模型 | 配置 | 每个任务的 token 数（基线） | 每个任务的 token 数（+记忆） | 额外开销 |
|---|---|---|---|---|
| DeepSeek-V3.2 | 完整指南集 | 148K | 263K | +78% |
| gpt-oss-120b | 完整指南集 | 110K | 166K | +51% |
| gpt-oss-120b | 精选检索 | 110K | 116K | +5% |
表 1. 每个任务的平均 token 使用量，跨智能体步骤累计，与无记忆基线对比。
两点结论：
精选检索让成本保持在基线附近。对于较弱的模型，在准确性上精选胜出，在成本上它也胜出——两全其美（gpt-oss-120b 的 TGC 提升 +16.1 个百分点，token 只增加了 5%）。更好的性能在这里并不需要更高的推理成本。
记忆不会让推理循环失控。DeepSeek 在带记忆和不带记忆的情况下运行的 ReAct 步骤数大致相同（平均约 18–19 步），所以增加的成本来自输入 token 的膨胀，而不是更长的推理轨迹。
生产环境中真正的效率杠杆是提示词缓存：指南集中静态部分在各步骤间完全一致，可以被缓存，从而大幅降低实际成本。缓存感知的提示词设计——保持共享指南集前缀稳定，使其始终可被缓存——值得投入工程精力。我们还推测上下文窗口大小也起作用：上下文窗口更大的模型可能更有效地吸收完整指南集，而上下文较小的模型则更受益于检索，使注入内容保持紧凑。我们尚未开展隔离这一因素的受控实验。
记忆应当被校准，而非仅仅累积
经验教训不是把智能体学到的一切都给它，而是给它它实际能用的经验量。
对于弱模型，这意味着一个紧凑的核心加上几条任务特定的经验——而这恰好也是最经济的选择。
对于有余力的强模型，这意味着保留完整指南集，并通过提示词缓存在生产中保持成本可控。
对于已饱和的模型，这意味着在剩余失败模式被更好理解之前，不额外占用上下文。
收益在各方面都是实实在在的——自动、无泄漏、无需人工标注——但前提是剂量匹配模型。
下一步
这是起点，而非终点：
学习型选择器。我们当前的检索按余弦相似度对指南排序，而我们已经证明这并不能完美预测哪些指南对给定任务有帮助。基于结果信号训练的选择器是自然的下一步。
极弱模型的记忆。低于最低能力基线时，自蒸馏缺乏信号。针对极弱模型的教师蒸馏记忆是我们正在探索的另一个独立问题。
超越 AppWorld。这些结果在 AppWorld 上得到验证——这是一个严谨的多步骤基准，但只是单一基准。更广泛的智能体基准和真实部署正在进行中。
隔离上下文窗口。如上所述，我们希望进行受控实验，将上下文窗口大小与原始能力分离开来。
可以试试 ALTK-Evolve 库——它包含了这里用到的抽取、整合和检索流水线——或者阅读完整的技术报告，了解完整方法和消融实验。
附录：理解各项指标
AppWorld 任务由两项指标评分，均以百分比形式报告（越高越好）：
TGC——任务目标完成率。智能体完整且正确完成的单个任务所占的比例。这是衡量“任务是否真正完成”的核心指标。
SGC——场景目标完成率。这是一个更严格的“全有或全无”指标。每个场景包含同一任务的多个变体（相同请求但数据、措辞或边界条件不同）。只有当智能体在每个变体上都成功时，SGC 才将该场景计为通过。它衡量的是可靠性——一个大多数时候能解决任务但在某个变体上失败的智能体，在 TGC 上能得分，但在 SGC 上则不能。
Vatche Isahagian Vatche Follow
Gaodan Fang gaodan-fang Follow
Jayaram Radhakrishnan jayaramkr Follow
Punleuk Oum illeatmyhat Follow
Ashwath Vaithinathan Aravindan ashwath-vaithina Follow
Evelyn Duesterwald evduester Follow
G Thomas gsthomasx Follow
Vinod Muthusamy vinodmut Follow
Merve Unuvar mrvnvr Follow
In our previous post, we compared ALTK-Evolve with ACE and showed that how you deliver an agent's self-distilled guidelines — a few retrieved per task vs. the whole set injected — drives both accuracy and cost. This post steps back to the question that comes before it: how much should you give it? Equipping an agent with agentic memory sounds simple: distill lessons from its past work, put them back in context, and more experience should mean better performance. It doesn't always work that way. When we scaled the evaluation to eight models — from a 30B dense model to frontier proprietary systems — one finding stood out:
Agentic memory is not a feature you switch on. It's a dose you calibrate to the model.
TL;DR
ALTK-Evolve lets an agent learn from its own past trajectories: distilling reusable guidelines and injecting them back at inference time, with no weight updates and no human annotation.
The right dose differs by model tier: strong models with headroom want the full guideline set, weaker models do best with a compact core plus per-task retrieval, and saturated models show no measurable gain.
Curated retrieval can be both the most accurate and the cheapest option: gpt-oss-120b gained +16.1pp task completion at only +5% tokens — and prompt caching keeps even the full guideline set affordable in production.
The Key Insight: Dosage Depends on Capability
Not every model benefits from the same amount of memory. Across eight models spanning the capability spectrum, we saw three recurring patterns:
Strong models with headroom want the full guideline set — every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set.
Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach — while the full guideline set gained less and cost ~50% more tokens.
Already-saturated models show no measurable gain. We call this the saturated pattern — the label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs.
What puts a model into one pattern rather than another isn't simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands, and separating those factors is ongoing work. The practical takeaway holds either way: the right dose of memory depends on the model, and we can calibrate it.
Learning happens around the model, not inside it
"Memory" here doesn't mean replaying a past transcript. It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories. The loop is straightforward:
The agent attempts tasks and produces trajectories.
ALTK-Evolve extracts behavioral guidelines from both its successful and unsuccessful runs.
It consolidates those guidelines into a reusable set.
At inference time, the agent receives either the full guideline set or a task-relevant selection of it.
No model weights are updated. The learning loop changes the guidance available to the agent, not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested.
Results Across the Spectrum
We evaluated on AppWorld — 585 multi-step tasks (168 testnormal + 417 testchallenge) across 9 simulated apps (calendars, messaging, payments, and so on). Tasks are scored two ways: whether the agent fully completes each task (TGC — Task Goal Completion) and whether every variant of a scenario passes (SGC — Scenario Goal Completion, a stricter, all-or-nothing bar). Full definitions are in the appendix.
The three configurations we compare
Because the confusing part of any memory study is what's actually in the context window, we define the configurations up front.
Both memory configurations draw from the same guideline set, mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it.
| Configuration | What's in the agent's context |
|---|---|
| Baseline | No memory — the agent as shipped. |
| Full guideline set | Every mined guideline, injected on every ReAct step. |
| Curated retrieval | A fixed, high-confidence core of those same guidelines plus a few task-relevant ones retrieved for each task (a fixed portion + a variable portion). |
The number of guidelines a model mines depends on its own capability, so we report configurations by strategy — "full guideline set" vs. "curated retrieval" — rather than by raw counts, which aren't comparable across models.
The three patterns, in one view
Representative models from the eight-model sweep, measured by task completion (TGC) on testnormal:
Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test\normal for baseline vs. the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below.
The figure plots TGC to keep it readable; the table adds the stricter SGC metric, where the gains are often larger:
| Model | Pattern | Baseline TGC / SGC | Best-memory TGC / SGC | Best config | Δ TGC | Δ SGC |
|---|---|---|---|---|---|---|
| gpt-oss-120b (117B MoE) | Weak / selective | 39.9 / 21.4 | 56.0 / 37.5 | curated retrieval | +16.1 | +16.1 |
| DeepSeek-V3.2 (671B MoE) | Strong w/ headroom | 79.8 / 64.3 | 89.3 / 80.4 | full guideline set | +9.5 | +16.1 |
| Claude Opus 4.6 | Strong w/ headroom | 90.5 / 87.5 | 94.6 / 94.6 | full guideline set | +4.1 | +7.1 |
| GPT-5.5 | Strong (near-ceiling) | 92.3 / 82.1 | 95.2 / 89.3 | full guideline set | +2.9 | +7.2 |
| GLM-5 (745B MoE) | Saturated | 87.5 / 80.4 | 87.5 / 80.4 | full guideline set | 0.0 | 0.0 |
Reading the SGC column, the stricter metric usually moves more than TGC — DeepSeek's SGC jumps +16.1pp against a +9.5pp TGC gain — because good guidelines especially help an agent clear every variant of a scenario, not just the average case. And the effect doesn't disappear at the top of the range: GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively. Memory keeps paying off as long as a model has a remaining failure mode to target.
The Cheapest Memory Strategy Can Also Be the Best
A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn. Here's what we observed:
| Model | Config | Tokens/task (baseline) | Tokens/task (+ memory) | Overhead |
|---|---|---|---|---|
| DeepSeek-V3.2 | full guideline set | 148K | 263K | +78% |
| gpt-oss-120b | full guideline set | 110K | 166K | +51% |
| gpt-oss-120b | curated retrieval | 110K | 116K | +5% |
Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline.
Two takeaways:
Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost — the best of both worlds (+16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require more inference cost.
Memory doesn't blow up the reasoning loop. DeepSeek runs about the same number of ReAct steps with memory as without (≈18–19 on average), so the added cost is input-token inflation, not longer trajectories.
The real efficiency lever in production is prompt caching: the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially. Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for. We also hypothesize that context-window size plays a role: models with larger windows may absorb the full guideline set more effectively, while smaller-context models benefit more from retrieval that keeps injected content compact. We have not yet run controlled experiments isolating this factor.
Memory Should Be Calibrated, Not Merely Accumulated
The lesson isn't to give an agent everything it has learned. It's to give it the amount of experience it can actually use.
For weak models, that means a compact core plus a few task-specific lessons — which, conveniently, is also the cheapest option.
For strong models with headroom, it means preserving the full guideline set, kept affordable in production via prompt caching.
For saturated models, it means spending no extra context until their remaining failure modes are better understood.
The gains are real across the board — automatic, leakage-free, and requiring no human annotation — but only when the dose fits the model.
What's Next
This is a starting point, not the finish line:
A learned selector. Our current retrieval ranks guidelines by cosine similarity, which we've shown doesn't perfectly predict which guidelines help a given task. A selector trained on outcome signal is the natural next step.
Memory for very weak models. Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem we're exploring.
Beyond AppWorld. These results are validated on AppWorld — a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress.
Isolating context window. As above, we want controlled experiments that separate context-window size from raw capability.
Try the ALTK-Evolve library — which includes the extraction, consolidation, and retrieval pipeline used here — or read the full technical report for the complete method and ablations.
Appendix: Understanding the Metrics
AppWorld tasks are graded by two metrics, both reported as percentages (higher is better):
TGC — Task Goal Completion. The share of individual tasks the agent completes fully and correctly. This is the headline "did it get the job done" number.
SGC — Scenario Goal Completion. A stricter, all-or-nothing metric. Each scenario bundles several variants of the same task (the same request with different data, phrasing, or edge conditions). SGC counts a scenario as passing only if the agent succeeds on every variant. It measures reliability — an agent that solves a task most of the time but fails on one variant scores on TGC but not on SGC.