Abstract

Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool routines, and learned skills across sessions. Yet whether retained experience actually improves them over time has not been systematically tested. We introduce PAST-Bench, a benchmark designed to isolate this question. Each agent runs through ordered sequences of fresh-session tasks under matched conditions that turn retained experience on and off. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update. We report both later-task gains and whether those gains follow the intended save, retrieve, and update pathway. Across seven base models and four agent frameworks, improvement is real but uneven across capabilities. Agents with the same headline gain can differ markedly in whether that gain is supported by evidence of the intended pathway. Guided by these findings, we develop Hermes+, which extends Hermes with five targeted interventions across stages of the agent loop. Hermes+ raises the average gain from retained experience and provides clearer pathway evidence, with its strongest improvement on tasks requiring outdated state to be replaced, although the effect remains capability- and model-dependent. Together, PAST-Bench and Hermes+ provide an evaluation and diagnostic foundation for studying how persistent agents can progress from retaining experience to systematically improving through it.

1 Introduction

Recursive self-improvement (RSI) concerns the ability of an AI system to use experience generated through its own operation to improve its future capabilities (wang2026openclaw; ren2026selfimprovements; lee2026recursive; qu2024recursive; yin2025godel). While stronger forms of RSI may eventually involve modifying model parameters, learning algorithms, or agent architectures, a more immediate and operational layer is already emerging in personal AI agents (wang2026openclaw; gao2025survey; sarukkai2025selfgenerated). Personal AI agents now persist across sessions. They read messages, operate over user workspaces, call tools, and accumulate files, memories, skills, and session histories over days and months (openclaw2026; hermes2026). Agent frameworks such as Hermes (hermes2026) and OpenClaw (openclaw2026) treat persistent workspaces, memories, skills, and tool execution as first-class runtime components, while memory-layer systems such as Mem0 (mem02026) and LangGraph (langgraphdeepagents2026) provide the substrate: editable memory, interaction-derived facts, temporal knowledge graphs, and procedural skill files. In these systems, user interactions are no longer merely transient context; they can become experience that changes the agent’s future behavior.

Personal agents thus provide a natural, user-grounded testbed for learning from experience. Before an agent can recursively improve the mechanisms by which it learns, reasons, or acts, it must first close a more basic loop: identifying useful experience, preserving it beyond the current session, retrieving it when relevant, applying it correctly, and revising it when it becomes outdated (xu2026amem). This shifts the unit of evaluation. The relevant question is no longer whether an agent solves the current task, but whether it becomes better at serving the same user across future ones—retaining durable preferences, reusing prior workflows, and revising stale information (buening2026aligning). We call this capability online self-evolution: a personal agent changes its future behavior by reusing experience accumulated during prior interactions, without model retraining (xia2025agent0; ou2025symbolic), prompt optimization (khattab2024dspy; yuksekgonul2025optimizing), or long-context adaptation (agarwal2024many). Online self-evolution is not RSI in its full form, but it provides a concrete behavioral and infrastructural substrate on which stronger forms of recursive improvement can be built (zhang2026memrl; fang2025comprehensive).

Accumulating experience does not guarantee improvement. An agent may store the wrong evidence, retrieve irrelevant memory, reuse brittle procedures, or apply stale state to a new task. Evaluating self-evolution is thus a performance-attribution problem: if later-session performance improves, the gain might come from retained experience—or from the base model, runtime, prompt, retrieval shortcuts, task difficulty, or scoring noise. Current benchmarks cannot make this distinction. Interactive agent benchmarks (liu2024agentbench; koh2024visualwebarena; drouin2024workarena; xie2024osworld; merrill2026terminalbench; mialon2023gaia; zhang2026clawbench) reduce evaluation to a one-shot per-task score on a fresh session, not a trajectory. Memory and skill benchmarks (wu2025longmemeval; maharana2024evaluating; li2026skillsbench) test individual ingredients of persistence in isolation, without matched controls that disentangle retained experience from base-model and runtime contributions.

To this end, we introduce PAST-Bench, a performance-attribution benchmark built around this question (Figure 1). The unit of evaluation is the agent’s trajectory through a task family rather than a one-shot per-task score: earlier episodes give the agent an opportunity to save reusable experience, later episodes test whether it is reused, and matched control episodes strip persistence so any later-task gain can be read against a no-persistence baseline. The current suite contains 26 scenarios and 204 episodes spanning four capabilities— memory (5/41), procedural reuse (8/64), information gathering (6/48), and update (7/51)—each targeting a distinct demand on persistent state. Holding model, task family, and evaluation interface fixed while toggling persistence makes the with/without gap directly comparable; saved artifacts and execution traces then reveal whether the agent actually wrote, retrieved, applied, or revised the state it was supposed to.

Contributions.

We address this gap with a benchmark, diagnostic study, and new agent framework.

(1) PAST-Bench: a benchmark for self-evolving personal agents (Section 3). A trajectory-level performance-attribution benchmark: 26 scenarios and 204 episodes across four capabilities, with matched persistence-on/off controls and trace-level evidence enabling per-stage diagnosis of where retained experience helps.

(2) Diagnosing self-evolution failures (Section 4). Across seven models and four agent frameworks, persistence gaps vary sharply by capability, and agents that tie on the task-score gap can still differ substantially in mechanism evidence (e.g., Hermes vs. nanobot: vs. at the same )—a discrepancy invisible to one-shot scoring.

(3) Hermes+: a new agent framework baseline (Section 4). Hermes+ extends Hermes with five runtime mechanisms, one per stage of the agent loop (Plan, Render, Route, Gate, Close), and serves as a new reference baseline that raises the reported means on both axes (: , Mech: ), with super-additive composition on Update (, well above any single mechanism: closeout alone , retrieval gate alone ). The Overall difference is smaller than run-to-run variation. We modify Hermes rather than other popular agent frameworks as it is the only framework in this set that exposes the agent loop without a pre-instantiated persistence stack, which is the property required for clean mechanism-by-mechanism ablation (Appendix C). The others are reported as off-the-shelf baselines.

2 Related Work

Prior work on agent evaluation falls into three groups, distinguished by the unit at which evaluation occurs. PAST-Bench departs from all three by grading an episode sequence within a task family rather than a single task instance, asking whether state created in earlier episodes is reused in later ones. Appendix F contains an extended account of related work.

Interactive and trajectory-level evaluation.

Interactive agent benchmarks (liu2024agentbench; koh2024visualwebarena; drouin2024workarena; xie2024osworld; merrill2026terminalbench) score complete agent stacks on isolated task instances, conflating base model capability, prompting, tool policy, and any retained experience into a single number. Trajectory-grading benchmarks (ma2024agentboard; he2025traject; li2026atbench) push further by scoring the action sequence within a task. PAST-Bench grades whether state produced in earlier tasks is reused across later tasks of the same family; within-episode trajectory evidence is an input to this attribution, not the outcome metric.

Memory, skill, and architectural mechanisms.

A second line of work evaluates specific persistence mechanisms in isolation: long-horizon conversational memory (wu2025longmemeval; maharana2024evaluating), curated or self-generated skills (li2026skillsbench; yang2026skillopt; ouyang2026skillos), and architectural choices (bogavelli2025agentarch). These works isolate one substrate but do not test whether retained experience improves later executable tasks under family-level controls. PAST-Bench uses cold, learning, evaluation, and control episodes to localize improvements to a specific persistence decision.

Mechanism attribution under contamination.

Recent critiques caution that benchmark scores can reflect familiarity with benchmark artifacts rather than transferable problem solving (liang2025swebenchillusion; berkeleyRDI2026brokenbenchmarks). The concern applies directly to self-evolution evaluation: later-task gains can be real score gains yet not caused by the persistence mechanism a framework claims credit for. PAST-Bench separates outcome gains from trace-level mechanism diagnosis and uses matched persistence-on vs. persistence-off controls to check whether the gap is robust to controls for task, model, and runtime. Accordingly, cross-session retention, a well-formed tool trajectory, or a self-generated artifact alone is not evidence of beneficial cross-episode reuse.

Table 1 situates PAST-Bench against representative agent benchmarks along four methodology axes: cross-session retained experience, fixed-framework model comparison, fixed-model framework comparison, and trajectory-level diagnostics beyond one-shot task success. Existing benchmarks cover strict subsets; PAST-Bench is the first to support all four jointly, which retained-experience attribution requires.

媒体内容 · 前往原文查看

Benchmark	Retained experience	Model comparison	Framework comparison	Trajectory diagnosis
GAIA(mialon2023gaia)	✗	✗	✗	✗
AgentBench(liu2024agentbench)	✗	✓	✗	✗
VisualWebArena(koh2024visualwebarena)	✗	✓	✗	✗
WorkArena(drouin2024workarena)	✗	✓	✗	✗
OSWorld(xie2024osworld)	✗	✓	✗	✗
LongMemEval(wu2025longmemeval)	✓	✓	✗	✗
LoCoMo(maharana2024evaluating)	✓	✓	✗	✗
SkillsBench(li2026skillsbench)		✓
AgentBoard(ma2024agentboard)	✗	✓	✗	✓
PAST-Bench	✓	✓	✓	✓

3 PAST-Bench

Current persistence-aware agent benchmarks fall into one of two regimes. The first keeps prior content visible to the model, either in a long context window (maharana2024evaluating; letta2025benchmarking) or by incremental injection into a growing dialogue history (hu2025evaluating). The second runs sequential task streams in which state propagates across tasks without interruption (wei2025evo; zheng2025lifelongagentbench). Both regimes conflate persistent learning with in-context propagation. PAST-Bench instead evaluates online self-evolution under a strict context-clearing protocol. Its evaluation unit is a task family: an ordered sequence of fresh-session episodes that share a latent rule (memory), a reusable artifact (procedural reuse), a correction (update), or a pre-seeded reference (information gathering), with the framework’s volatile context wiped between episodes. Any improvement of a later episode over an earlier one must therefore flow through the persistent substrate—memory store, skill file, playbook, or edited rule—not through residual prompt overlap.

3.1 Benchmark Construction

PAST-Bench targets four core capabilities of online self-evolution, each requiring retention across sessions and active reuse in later ones: Memory, Procedural Reuse, Information Gathering, and Update. For each capability, we curate a diverse set of carefully designed task families. Each task family is an ordered sequence of fresh-session episodes sharing a latent rule (memory), a reusable artifact (procedural reuse), a correction (update), or a pre-established reference whose retrieval must be triggered at the right moment (information gathering). For the first three capabilities, earlier episodes expose information, a procedure, or an updated value to be retained; for information gathering, the reference is preseeded before the family begins. In every case, later episodes test whether the agent reuses or consults the relevant state without restating the decisive rule. Each family also contains control episodes: no-retention controls that remove the earlier state, distractor controls that inject irrelevant or superficially similar information, stale controls that expose obsolete memories, and wrong-mechanism controls that surface incorrect skills or the wrong evidence source. The distribution of families across capabilities and scenario groups is shown in Fig. 2. The per-capability breakdown is reported in Table 7 in Appendix A.1. Appendix A.3 presents detailed trace-backed examples in Figures 4–8.

Capability 1: General-knowledge memory—can the framework retain and look up an everyday clause? Memory families isolate the declarative pathway (sumers2023cognitive; squire1992declarative) for casual, user-facing facts (preferences, constraints, one-line policies, exceptions, prior-case decisions). Each family seeds a single read-mostly clause and never revises it; evaluate episodes succeed iff the agent recovers and applies it as a one-shot lookup-and-apply, with the trigger wording removed from the prompt. When to query is deferred to Capability 3; revising a clause is deferred to Capability 4.

Capability 2: Professional procedural reuse—can the framework retain and re-execute a multi-step technical workflow? Procedural-reuse families isolate the imperative pathway (sumers2023cognitive; anderson2014atomic) for domain-specific, technical routines—SOPs, playbooks, build/deploy pipelines, incident-triage flows, induced engineering workflows (hong2023metagpt). Where Capability 1 tests value lookup, Capability 2 tests ordered execution with the right tool composition; order errors, skipped steps, and wrong-tool substitutions are graded as failures. Procedure revision is deferred to Capability 4, so this capability isolates first-time procedure formulation.

Capability 3: Information gathering—given that the answer is already in the substrate, does the framework consult it at the right moment? The relevant artifact is pre-seeded into the runtime’s persistence layer (long-term memory, registered skills, indexed transcripts, or home-state fixtures) before the family begins. The test is not retention but whether the agent proactively retrieves under noisy context; each family also plants a generic default that yields a plausible-but-wrong answer if used in place of retrieval.

Capability 4: Update—can a second write override a first one without leaking the first? Update families begin with an existing entry (stale fact, old rule, outdated SOP, or temporary exception) and then deliver an authoritative second write; evaluate episodes score whether the new state is used and the old state does not leak. Seven families cover fact correction, global and scoped rule migration, temporary-exception expiry, incident and error-correction SOP patching, and recall-then-modify of a prior session artifact. Figures 4–8 in Appendix A.3 give one representative example per capability.

3.2 Evaluation Pipeline

Episode roles.

Every task family is an ordered sequence of fresh-session episodes playing one of four roles. Cold episodes measure first-contact behavior before any persistence can exist. Learn episodes (and, in Update families, an Update episode) deposit the target (a clause, procedure, or correction) into the persistence substrate. Evaluation episodes probe reuse of that state in a later fresh session with the trigger wording removed. Control episodes check that any gain cannot be explained by prompt shortcuts, surface-memorization, stale reuse, or writes to the wrong substrate.

Persistence and the matched ablation.

By persistence we mean benchmark-managed access to state produced or modified by earlier episodes in the same family: memory records, skills, profile entries, session-history indices, saved artifacts, and home-state fixtures. Each evaluation episode is graded under a matched ablation. The w/o-evolve condition denies the runtime any access to family-produced state; the w/-evolve condition permits it. The two runs share the same prompt, grader, tool stack, and seed, so any score gap is most plausibly attributable to the persistence layer rather than to model variance or task drift. We treat this as a strong design control rather than a causal proof; the mechanism-evidence score below provides a complementary substrate-level check. Cold scores are reported for calibration and headroom analysis but are not the persistence-off baseline; the matched w/o-evolve condition is.

Reported quantities.

The primary metric for each family is the self-evolution gap , where is the within-family mean of the per-episode task score . The capability-level is the macro-average of over families. We accept as evidence of self-evolution only when it clears the upper bound set by the family’s control episodes for shortcut, surface-memorization, stale-reuse, and wrong-mechanism explanations. Alongside the gap, PAST-Bench reports mechanism evidence: memory writes and reads, skill creation and patching, session-search calls, artifact diffs, and update-correctness signals. An agent that writes artifacts but never reads them, or succeeds through the wrong substrate, should not earn credit on alone. The task-score definition (Equation 1) and mechanism-evidence aggregation (Equation 2), together with control bounds and per-family rubrics, appear in Appendix B.

4 Experiments

We first describe the controlled self-evolution setting (Section 4.1), report main results across models and agent frameworks (Section 4.2), use those results to diagnose where self-evolution breaks down and introduce Hermes+ (Section 4.3), and finally check whether the diagnosed fixes show up in mechanism ablations and on other base models (Section 4.4).

4.1 Experimental Setup

Each PAST-Bench task family is evaluated under paired persistence conditions: persistence-off (no retained state from the learning episode) and persistence-on (the agent may reuse memory, skills, profile state, or session history). We report the persistence-on score and the family-balanced gap . This gap measures whether retained experience improves later-task performance. Per-episode computational costs (tokens, wall time) are reported in Table 12 in Appendix D.6.

alone is not sufficient evidence of self-evolution. We also report a mechanism-evidence score based on saved artifacts and runtime telemetry (see the Mechanism-Evidence Score subsection and Equation 2 in Appendix B), asking whether the improvement used the intended persistence pathway. Throughout this section, task score and measure behavior; mechanism evidence supports attribution.

4.2 Main Results

媒体内容 · 前往原文查看

Model	State	Memory	Procedural	Info.	Update	Overall	Mech.
GLM-5.1	w/o evolve						–
	w/ evolve
							–
Kimi K2.6	w/o evolve						–
	w/ evolve
							–
DeepSeek-V4-Pro	w/o evolve						–
	w/ evolve
							–
MiniMax-M2.7	w/o evolve						–
	w/ evolve
							–
GPT-5.4	w/o evolve						–
	w/ evolve
							–
Claude Sonnet 4.6	w/o evolve						–
	w/ evolve
							–
Claude Opus 4.6	w/o evolve						–
	w/ evolve
							–

Table 2 fixes the agent framework to Hermes and varies the base model. The leftmost column records the evolution state: persistence-off (w/o evolve), persistence-on (w/ evolve), or their difference (). Mechanism score is reported separately so that higher accuracy is not confused with evidence-aligned self-evolution. Table 3 fixes the model to MiniMax-M2.7 and varies the agent framework, separating model effects from runtime effects.

媒体内容 · 前往原文查看

Agent	Memory	Procedural	Info.	Update	Overall	Mech.
nanobot
ZeroClaw
Agent-Zero
Hermes
Hermes+ (Our framework)

Self-evolving agent frameworks are robust across base models, but the subtasks that benefit most depend on each model’s strengths.

Every base model in Table 2 gains from persistence (Overall from to ), so the runtime carries persistence value across architectures. Where each model concentrates that gain, however, varies sharply with its own profile (capability subscripts on the rows): GPT-5.4 spreads its movement evenly across Memory () and Update (); GLM-5.1 places nearly half of its movement on Update (); Kimi K2.6 places nearly half on Memory (); DeepSeek-V4-Pro and Claude Sonnet 4.6 sit between these extremes. The capability the model already excels at is also where retained experience helps it most, which is why a single Overall tells the wrong story: the four-capability decomposition is what reveals the model-specific strengths.

Advanced agent frameworks show diverse strengths across task subtypes.

Each framework in Table 3 concentrates its movement on a different subtype (capability subscripts): ZeroClaw lifts Memory ( of its movement) but loses ground on Procedural; nanobot puts on Update yet barely improves Memory; Agent-Zero regresses on three of four capabilities. Hermes is the only baseline framework that moves all four capabilities upward; Hermes+ lifts Memory, Information Gathering, Update, and Overall but shows a small Procedural dip (). Even with this dip, Hermes+ shifts more of its gain onto Update ( vs. Hermes’s ) without sacrificing Memory. The mechanism-evidence score moves with this shape: nanobot and Hermes both reach , but nanobot earns it from a single capability with no consistent write-then-read trace, dropping its Mech to against Hermes’s . The same headline can hide two completely different ways of getting there (Figure 10 in Appendix D.3 plots the frontier).

4.3 Diagnosis-Driven Design: Hermes+

Section 4.2 uses PAST-Bench as a diagnostic tool. Table 2 fixes the framework and varies the model, showing that different models gain on different capabilities. Table 3 fixes the model and varies the framework, showing that the same task-score gain can come with different mechanism evidence. We then inspected low-gain and uneven-gain traces. The failures were concrete and fell into five disjoint categories: plans were drafted without first consulting saved state, saved facts appeared in the wrong form, learned procedures stayed outside the skill library, stored evidence was skipped before action, and corrected state failed to reach the next fresh session. Each category is the responsibility of a single loop stage, and we treat each as the responsibility of one mechanism so that the mechanisms can be enabled or ablated independently of one another.

We choose Hermes for the intervention study because it is the strongest controlled baseline among the existing agents. In Table 3, Hermes is the only non-Hermes+ framework with positive on all four capabilities. It also ties nanobot for the best baseline Overall () while showing stronger mechanism evidence ( vs. ); ZeroClaw and Agent-Zero are lower on Mech ( and ). Hermes already exposes the persistence surfaces PAST-Bench probes: memory records, user-profile state, skills, and session history. This lets us keep the model, tasks, grader, and substrate fixed while changing only the runtime decisions. Here, runtime means the decisions that choose when to read, write, or trigger the persistence surfaces.

The plan does not condition on saved state (cross-cutting). A failure that recurs across all four capabilities: even when the relevant state has been correctly written and is retrievable, the agent’s plan is often drafted without first consulting it, so a draft action gets committed before any saved binding, skill, or rule is brought into the planning context. This is upstream of the ability-specific failures below, since an unconditioned plan can override correctly stored state regardless of how that state is shaped. We add a plan-time consultation check (E1). E1 sits at the planning stage of the loop and, before drafting any risky or recall-dependent action, requires the agent to consult whichever persistent state the runtime currently exposes and to condition the plan on it. E1 reads the typed schema produced by E2 when E2 is active and reads the native Hermes memory records otherwise, so it operates as a stand-alone plan-time gate that does not require any of the other four mechanisms to be enabled.

The saved memory is hard to reuse in a fresh session (Memory). Memory families test declarative facts such as preferences, constraints, and corrected values. Hermes can store these facts, but the storage design is too loose: current and stale notes can sit together, scope is implicit, and the next session may not see a clear valid clause to apply. We store memory as typed bindings and render only the valid one (E2). E2 writes each memory with type, scope, entity, current value, superseded value, and expiry, and at render time surfaces only the in-scope current binding to the next session, suppressing the superseded entries that would otherwise compete with it.

The solved workflow is not saved as an executable procedure (Procedural). Procedural families test ordered execution, not one-shot fact lookup: the agent must reopen a workflow and follow its steps. Hermes traces show successful learn episodes whose SOP remains in transcript text or splits into near-duplicate notes. Later episodes ask for the saved procedure, but there is no ranked skill to open and execute. We save procedures as ranked, patchable skills (E3). E3 writes a skill with an applicability condition and ordered steps, ranks saved skills by query relevance, and directs the model to patch the closest existing skill when the workflow changes.

The agent acts before checking stored evidence (Information Gathering). Information-gathering families preseed the needed evidence into memory or session history. The failure is trigger timing: under noisy prompts, Hermes may answer from visible context, refuse to guess, or take an irreversible action before calling the persistence channel. We require retrieval before recall-dependent actions (E4). E4 blocks a draft answer when the task depends on prior state and no persistence read has occurred, then requires a read from the relevant channel.

Old persistent evidence remains active after correction (Update). Update families provide a second authoritative value and then test whether the agent uses it in a fresh session. Hermes traces often record the correction inside the current session, while the next episode reads an older artifact or an unstructured transcript fragment. The result is a stale answer after the user has already corrected it. We make the new persistent evidence overwrite the old one (E5). E5 extracts the final binding key or updated rule at episode close, writes it as the new authoritative artifact in place of the prior value, and flushes it synchronously to the persistent store, so the next fresh session retrieves only the corrected value rather than reading the older artifact alongside it.

Together, the cross-cutting plan-time check and the four ability-specific fixes map one-to-one onto the five colored runtime insertion points in Figure 3. Each mechanism is wired as an independent drop-in at its own loop stage, with the remaining stages held at the Hermes default whenever a mechanism is run in isolation. Figures 4–8 in Appendix A.3 give trace-backed case studies for the corresponding mechanisms.

4.4 Mechanism Evidence and Generalization

Table 4 isolates the contribution of each mechanism. Each non-final row adds a single mechanism on top of Base Hermes, leaving all other loop stages at their defaults; rows are therefore not cumulative. The final row, Hermes+ (full), turns on all five mechanisms simultaneously. All settings share a fixed model (MiniMax-M2.7), task set, and grader. For each capability we report two numbers: the persistence-on score (w/) and the persistence-on/off gap ().

媒体内容 · 前往原文查看

	Memory		Procedural		Info.		Update		Overall
Setting	w/		w/		w/		w/		w/
Base Hermes
+ planning guidance (E1)
+ memory binding (E2)
+ skill lifecycle (E3)
+ retrieval gate (E4)
+ closeout / flush (E5)
Hermes+ (full)

Single mechanisms support their target diagnoses.

E2 (Render) gives the highest Memory persistence-on score (), E3 (Route) the largest single-mechanism Procedural (), E4 (Gate) the largest Info (), and E5 (Close) the strongest single-mechanism Update (). Together, these results align with the failure-to-mechanism mapping in Section 4.3.

The full Hermes+ preserves overall task performance and has its clearest gain on Update.

Hermes+ (full) ties Base Hermes on Overall persistence-on score (), raises the reported Overall from to , and reaches the best Update score () and gap (), while its Procedural result declines slightly. Figure 9 provides the per-capability view.

媒体内容 · 前往原文查看

Full-minus-one persistence gaps
Setting
Base Hermes
Full Hermes+
w/o E1
w/o E2
w/o E3
w/o E4
w/o E5
Routing comparison
Measure	Full	w/o E2
No task-specific skill before evaluation	2/6	1/6
No task-specific skill read during evaluation	3/6	2/6

A focused Procedural diagnosis exposes mechanism interaction.

The full-minus-one rows in Table 5 show that removing E2 raises from to , whereas removing E3 or E5 lowers it to or . The routing rows link the E2 effect to more consistent task-specific skill creation and reuse. We treat this as a focused diagnostic rather than a full-benchmark estimate; Appendix D.2 provides a representative trace.

Hermes+ improves or preserves the Hermes baseline on most base models.

Swapping Hermes+ (tuned on MiniMax-M2.7) onto five base models matches or improves each model’s Hermes baseline on three of five configurations — MiniMax-M2.7 (), Claude Sonnet 4.6 (), and GPT-5.4 (flat at ). On its strongest pairing, Hermes+ + GPT-5.4 ties the benchmark’s highest configuration (, Mech ). DeepSeek-V4-Pro and Claude Opus 4.6 regress slightly, so the transfer result is positive but not uniform. Table 6 gives the full per-capability comparison.

The three analyses answer different attribution questions.

Table 4 tests isolated interventions, Table 5 tests mechanism interactions, and Table 6 tests cross-model transfer. Together they show why Overall alone is insufficient: target gains can coexist with regressions elsewhere, component effects can reverse in combination, and a runtime change need not transfer uniformly.

媒体内容 · 前往原文查看

	Memory		Procedural		Info.		Update		Overall		Mech.
Model	w/		w/		w/		w/		w/		Score
DeepSeek-V4-Pro
MiniMax-M2.7
GPT-5.4
Claude Sonnet 4.6
Claude Opus 4.6

Run-to-run variation tempers the aggregate comparison.

Across three MiniMax-M2.7 runs, the Overall gap is for Hermes and for Hermes+. The difference is smaller than the run-to-run variation, so we do not interpret it as a stable overall gain. The clearer mean shift is on Update (), although its also increases from to ; Appendix D.5 (Table 11) gives the full capability-level breakdown.

5 Conclusion

We introduced PAST-Bench, a performance-attribution benchmark that pairs persistence-on/off evaluations within task families and reports mechanism evidence alongside task scores, separating base-model, runtime, and retained-experience contributions to later-task performance. Experiments across seven models and four frameworks show that self-evolution is capability-specific and that similar persistence gaps can hide different persistence paths. Under MiniMax-M2.7, Hermes+ raises the reported mean Overall from to and Mech from to , with its clearest gain on Update; the Overall difference is smaller than run-to-run variation. The effect is not uniform across capabilities or base models, so we treat Hermes+ as a diagnostic scaffold rather than a universal improvement.

6 Future Work

PAST-Bench provides an initial foundation for attributing cross-session improvement to retained experience, but several directions remain open. First, future versions should broaden the ecological validity and temporal scope of the benchmark. The current task families are synthetically constructed and evaluated in isolation. An important next step is to incorporate human-authored and interaction-derived scenarios, longer task sequences, and settings in which experience accumulated in one family affects behavior in another. Such extensions would test whether persistent agents can maintain useful state over longer horizons, transfer experience across changing domains, and avoid interference among independently acquired memories, procedures, and corrections.

Second, the capability space should be expanded beyond memory, procedural reuse, information gathering, and update. These capabilities represent necessary foundations of online self-evolution, but they do not cover stronger forms of recursive improvement. Future benchmarks could evaluate whether agents acquire previously unavailable tool-use strategies, construct and revise long-horizon plans, coordinate experience across multiple agents, and improve the mechanisms by which they decide what to store, retrieve, verify, and update. This would help distinguish systems that merely reuse retained state from systems that improve their own learning and adaptation processes.

Third, future work should strengthen mechanism attribution. The current mechanism-evidence score measures consistency with an expected persistence pathway, rather than establishing causal necessity. A stronger evaluation could combine trace evidence with counterfactual interventions, such as deleting, replacing, or corrupting a candidate artifact and measuring the resulting behavioral change. It would also be useful to support multiple semantically valid persistence pathways, since different agents may encode the same experience as a memory, skill, structured artifact, or revised policy. Larger-scale human pathway annotations and uncertainty estimates would further improve the construct validity of mechanism-level evaluation.

Finally, the capability-specific and model-dependent behavior observed in Hermes+ suggests that persistence mechanisms should not be treated as uniformly composable. Future agents could learn to route experience dynamically across memory, skills, and session history, while detecting conflicts, redundancy, and stale state across these substrates. In particular, the interaction between structured memory rendering and procedural skill routing motivates adaptive mechanisms that decide not only when to read or write persistent state, but also which persistence surface should own a given piece of experience. Developing such mechanisms under explicit accuracy, latency, and token-cost constraints may provide a practical path from persistent agents that retain experience to agents that systematically improve through it.

Acknowledgments

Z Ding and Y Chen are supported by the U.S. National Science Foundation (NSF) under grants 2037026, 2313131, 2543755 and 2607613.

References

Appendix A PAST-Bench Benchmark Details

A.1 Family Taxonomy

Table 7 reports the full task distribution of PAST-Bench.

媒体内容 · 前往原文查看

Main Family	#Episodes	Sub-Family (Focus Area)	#Episodes
Memory	41	Preference Adoption	8
		Constraint Retention	8
		Weak-Trigger Preference Adoption	9
		Prior Case Recall	8
		Exception List Recall	8
Information Gathering	48	Release Decision Followup	8
		Ops Exception Desk	8
		Oncall Handoff Lookup	8
		Temporary Waiver Audit	8
		Change Freeze Followup	8
		Kappa Integration Review	8
Procedural	64	SOP Bootstrap 01	8
		SOP Bootstrap 02	8
		SOP Bootstrap 03	8
		SOP Bootstrap 04	8
		SOP Bootstrap 05	8
		SOP Bootstrap 06	8
		Latent Rule Induction	7
		Failure-to-Rule	9
Update	51	Fact Correction	8
		Rule Migration	8
		Temporary Exception Pollution	7
		Scoped Rule Migration	7
		SOP Patch 01	7
		SOP Patch 02	7
		Recall-then-Modify	7
Total	204

A.2 Task Construction and Quality Control

All 26 task families and 204 episodes are synthetic. No task contains data from real users. We first defined the four capabilities from common problems in human–agent interaction: keeping user-specific facts, reusing learned procedures, finding earlier evidence in noisy histories, and replacing outdated state. We then wrote rules for weak triggers, ambiguity, distractors, conflicting records, and transfer across fresh sessions.

The concrete families and episodes were generated from these rules with two model–agent pairs: Codex with GPT-5.4 and Claude Code with Claude Opus 4.6. The prompts, graders, and expected artifacts were generated in the same process. Existing benchmarks informed the high-level scenario taxonomy, but no task was copied from or adapted from another benchmark.

Three authors ran and checked the generated tasks. Each family was checked by at least one author. The check covered six points:

1. 1.

   the family and episode roles match the target capability;

2. 2.

   each prompt is complete and logically consistent;

3. 3.

   ambiguity and weak triggers are intentional;

4. 4.

   preferences and corrections stay consistent across episodes;

5. 5.

   distractors and controls do not reveal the retained item or create a shortcut; and

6. 6.

   graders and expected artifacts match the prompt, partial-credit rules, and stale-answer rules.

A.3 Trace-Backed Mechanism Case Studies

Each PAST-Bench family probes a distinct cross-episode dependency that a learn episode establishes and a later eval episode must recover: Constraint Retention (B1), Fact Correction (B2), SOP Bootstrap (B3), Exception List Recall (B4), and Failure-to-Rule (B5) generated by MiniMax-M2.7 under Hermes Agent. Figures 4–8 present one representative episode per family, contrasting an agent trace that fails to recover the dependency with one that does. The evaluation user message and final answer are reported; intermediate reasoning and tool outputs are abridged, with ellipses (…) marking omitted material. Each caption names the failure mode the family is designed to surface.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Appendix B Metric Definitions and Aggregation

This section specifies exactly how the two reported scores, task score and mechanism-evidence score (Mech), are computed from raw traces and aggregated to the family, capability, and benchmark levels.

B.1 Task Score

Each episode is graded by a task-specific grader that evaluates the agent’s trace (messages, tool calls, audit data) and produces three dimension scores:

• •

  Completion  : task-specific quality of the agent’s output. For action-oriented families, completion is computed from audit data (e.g., did the agent share with the correct recipients, update the correct tickets, output the correct facts). For open-ended families, an LLM judge evaluates the final output against a rubric. Each family defines its own grader; all graders return a value in .

• •

  Robustness  : recovery rate from tool-call errors, computed as follows. Let be the ordered sequence of tool dispatches in the episode. An error dispatch is any with HTTP status . Let be the set of distinct tool names that produced at least one error, and be the subset that were subsequently called successfully (i.e., the agent retried and recovered). The recovery rate is . As a floor, an agent that makes many successful calls despite some errors receives partial credit: , where . The robustness score is:

• •

  Safety  : binary gate for safety violations. A safety violation zeros out the entire score.

The per-episode task score combines these dimensions with fixed weights:

(1)

Each episode is run across three independent trials. Missing or crashed trials score .

B.2 Aggregation

Episode family.

Episodes within a family are grouped by bucket (baseline, learn, evaluation, control). The family-level evaluation score under persistence condition is the arithmetic mean of task scores across all evaluation-bucket episodes:

Family capability.

Capability-level scores macro-average over families: .

Capability overall.

The overall score is the mean of the four capability-level scores.

Self-evolution gap.

The per-family gap is ; the capability-level gap is ; the overall is the mean of the four capability-level deltas. The w/o baseline is the matched ablation (same prompt, grader, tools, seed; persistence stripped), not the cold-start score.

B.3 Mechanism-Evidence Score (Mech)

The mechanism score measures whether the agent used the intended persistence pathway, not just whether task scores improved. Intuitively, Mech means the agent completed the full expected persistence cycle (write retrieve correct application); Mech means the pathway was entirely absent.

B.3.1 Per-Episode Computation

Each episode specifies an expectation contract in its family YAML: the expected artifact type (memory, skill, or session_search), required keyword patterns, minimum write/read counts, and retrieval signals. The mechanism scorer compares the actual trace against this contract. Artifact quality is defined as follows:

Artifact quality .

Measures whether the agent wrote the correct persistent state. Computed as the mean of two sub-scores:

1. 1.

   Keyword hit rate: the fraction of expected rule keywords that appear in the saved artifact (memory entries or skill content).

2. 2.

   Count-delta score: whether the expected number of entries were created or updated. Formally, let be the observed count delta (e.g., number of new memory entries) and be the contract’s min_count_delta. The count-delta score is .

where includes only the components that are specified in the contract (e.g., if no keywords are required, only the count-delta score is used).

B.3.2 Family-Level Mechanism Score

The family-level mechanism score aggregates five sub-scores computed from the episode-level signals across the family’s learn and evaluation episodes:

(2)

• •

  Write precision (wp): average artifact quality across learn episodes. Measures whether the agent wrote the correct state during the learning phase.

• •

  Recall accuracy (ra): average content-correctness of evaluation episodes that used the expected retrieval signal. An episode contributes its grader-assigned content-correctness score if it fired the expected signal; otherwise it contributes .

• •

  Update correctness (uc): for episodes involving updates (learn-phase updates and evaluations), the mean of (a) stale-memory resistance (binary: if the artifact shows updates, changes, or additions; otherwise) and (b) content-correctness:

• •

  Retention horizon (rh): ratio of eval-far to eval-near task scores, measuring whether persisted state survives domain shift:

  A value of means the agent performs as well on distant evaluation episodes as on near ones; values below indicate decay.

• •

  Pollution rate (pr): fraction of written entries in learn episodes that are irrelevant or out of scope. Subtracted from in Eq. 2 so that lower pollution yields a higher score.

Capability and overall Mech.

 ; overall Mech is the mean of the four capability-level values.

B.4 Human Validation of the LLM Judge

We compare the open-ended LLM judge with independent human scores on 48 blinded samples, with 12 samples from each capability. The sample covers Hermes with different base models and MiniMax-M2.7 with different frameworks. Two authors scored each sample with the same rubric and evidence used by the judge. They did not see the judge score, model, framework, persistence condition, run, trace identity, or each other’s score. Judge–human agreement compares the MiniMax-M2.7 judge score with the mean of the two human scores.

媒体内容 · 前往原文查看

Audit set		Human–human		Judge–human
		Exact	Within 0.25	Within 0.25	Within 0.5
Four-capability audit	48	83.3%	97.9%	68.8%	91.7%

The two human scorers agree closely. Agreement between the judge and the human mean is useful but imperfect. We therefore use the LLM judge as a scalable grader with human validation, not as a substitute for human judgment. All benchmark runs use MiniMax-M2.7 as the judge with temperature 0 and a maximum output of 8,192 tokens. We do not vary the judge model or prompt in this study.

B.5 Sensitivity of Mechanism Evidence

We recompute Mech on archived Hermes traces from six base models. Changing one component weight from 1.0 to 0.8 or 1.2 gives Spearman correlations from 0.970 to 0.997. Agreement on whether Mech is at least 0.5 ranges from 98.8% to 100%. Removing keyword-based artifact-content credit gives a Spearman correlation of 0.868 and 90.7% threshold agreement. Requiring an explicit retrieval event before the final answer gives a Spearman correlation of 0.964 and 98.8% threshold agreement.

These tests preserve most rankings and threshold decisions. The keyword test causes the largest change, which shows that artifact content still matters to the score. Mech should be read as a stable pathway signal under these tested changes, not as causal proof.

Appendix C Agent Scope and Framework Details

C.1 Personal-Agent Frameworks

The intervention study requires a framework whose loop and persistence surfaces can be modified while keeping the model, tasks, and grader fixed. We evaluate three further framework snapshots through benchmark adapters. The exact source snapshots and adapters are frozen in the released artifact at revision 0b56a98. The adapters select the common model, expose task tools, and implement the matched persistence control; they do not add the five Hermes+ mechanisms. Thus the comparison preserves each framework’s agent loop, but is not a byte-for-byte default deployment.

We select the Hermes v2026.4.16 snapshot because its single-agent loop directly exposes the memory, user-model, skill, and session-search surfaces used in our evaluation. This allows the runtime decisions around those surfaces to be added, removed, and ablated while the underlying substrate remains fixed. We therefore modify Hermes (yielding Hermes+) and report the three other frameworks as adapter-standardized baselines.

Agent-Zero (agentzero2026)

As an external comparison point, we additionally evaluate Agent-Zero, a multi-agent framework that uses recursive sub-agent decomposition and includes built-in instrumentation for long-term memory, skill files, and inter-agent delegation. We do not add the five Hermes+ mechanisms. The adapter selects the evaluated model through Agent-Zero’s model configuration, exposes task tools through its native usr/tools path, and normalizes provider-standard tool arguments while retaining the recursive agent loop. Any of the evolve mechanisms we propose would semantically overlap with infrastructure Agent-Zero already provides (e.g., a parent agent already routes tasks through subordinate workers that read and write into a shared memory directory), so adding our mechanisms on top would not yield a clean ablation. The comparison instead asks how targeted, mechanism-by-mechanism augmentation of one substrate compares with Agent-Zero’s own integrated design under the same benchmark interface.

We run Agent-Zero’s recursive loop with a relaxed per-task wall-clock budget of 1200s (4× the per-task budget used for Hermes/Hermes+) to accommodate recursive sub-agent decomposition. The relaxed budget was chosen empirically: at the default 300s budget, a non-trivial fraction of Agent-Zero runs exceed wall-clock before completing a single user-facing task, due to the model-call multiplication inherent to its multi-agent design. We report the budget as a methodological footnote rather than a fairness adjustment. We agree that Agent-Zero is competitive on score given enough budget, and our purpose in including it is to characterize a different design point in the agent-framework design space, not to produce a head-to-head winner.

ZeroClaw (zeroclaw2026)

The reported ZeroClaw result uses the repository’s Python zeroclaw-tools companion rather than the Rust executable. The adapter runs its LangGraph tool loop with recursion limit 100 and supplies benchmark task tools, memory, and session search under the matched persistence toggle. It therefore represents a compact loop-based runtime, not a no-loop or stateless lower bound.

Including ZeroClaw tests whether the matched persistence protocol transfers across implementations with different orchestration and state-management choices. Because these frameworks differ in several respects, the comparison characterizes design points rather than isolating a single architectural component.

nanobot (nanobot2026)

We also considered nanobot, an “ultra-lightweight” personal-assistant framework, as a candidate substrate, but rejected it on the same grounds as Agent-Zero: Nanobot is minimal in lines of code, not in mechanisms. It ships with a token-budgeted memory subsystem, a skill marketplace (ClawHub), subagent dispatch, Cron scheduling, and MCP tool extension. Each overlaps one of the evolve mechanisms we study, so layering our modules on top would conflate our contribution with nanobot’s existing persistence stack.

A second reason is structural. Nanobot’s design center is operational deployment (channel plumbing, OAuth, streaming, multi-platform routing), not task-completion substrate. The components our augmentations target, namely the loop, the artifact store, and the inter-episode handoff, are reached only after several layers of channel- and provider-level abstraction, which precludes the mechanism-by-mechanism additions an ablation requires. Hermes, by contrast, directly exposes the loop and persistence surfaces needed for independent intervention toggles.

We therefore do not add the five Hermes+ mechanisms to nanobot — its built-in persistence stack would conflate them with infrastructure nanobot already provides. The adapter selects the provider, registers benchmark task tools, and invokes nanobot’s native AgentLoop and memory consolidation. We report it alongside Agent-Zero and ZeroClaw because it occupies a distinct point in the agent-architecture design space. Hermes remains the augmentation substrate.

C.2 Evaluation on General-Purpose Agents

The main experiments focus on personal agents because these systems are designed to keep user-specific state across sessions. We also test whether the protocol applies to general-purpose agents. These systems support a broad range of open-ended tasks and are not designed only for personal assistance.

We evaluate Codex CLI and Claude Code with MiniMax-M2.7. Both agents use the same tasks, graders, and matched persistence-on/off protocol as the main experiments. Results are means over three independent runs. Table 9 reports the persistence-on score and the matched gap .

媒体内容 · 前往原文查看

Agent	Memory	Procedural	Info.	Update	Overall
Codex CLI
Claude Code

Both general-purpose agents have positive matched gaps on all four capabilities. These results show that the PAST-Bench protocol can measure retained-state use outside personal-agent frameworks. They do not imply that the two systems are personal agents or that the result covers every general-purpose agent.

C.3 Support for Different Persistence Interfaces

The matched protocol only requires a way to turn access to retained state on and off. A black-box agent can therefore report Task Score and when this control is available. Mech requires observable persistence events. If an agent does not expose these events, Mech is unavailable. If it exposes memory, skill, or history events, a small adapter can map them to the benchmark event types.

Appendix D Additional Experimental Results

D.1 Mechanism Ablation Heatmap

Figure 9 visualises the per-capability persistence gap for every single-mechanism addition and the full Hermes+.

D.2 Procedural Routing Trace

The clearest example is PC03_latent_rule_induction_01. With full Hermes+, both learning sessions call skills_list but never call skill_manage. The evaluation sessions therefore have no DB-migration skill to reuse. Without E2, the first learning session creates a DB-migration skill. The next learning session reads it, and both evaluation sessions open it with skill_view. Removing E2 makes the skill destination clearer and raises the Procedural gap from to in this focused diagnosis.

D.3 Agent-Level Attribution Frontier

Figure 10 plots the fixed-model agent comparison on the two attribution axes reported by PAST-Bench.

D.4 Per-Family Paired Results

Table LABEL:tab:per_family_paired_results gives the full 26-family breakdown for Hermes and Hermes+ with MiniMax-M2.7. Values are means over three runs. The score is measured with persistence on, and is the matched persistence-on/off gap.

媒体内容 · 前往原文查看

	Hermes		Hermes+
Family	w/		w/
Memory
EP01_prior_case_recall	0.583	+0.315	0.650	+0.334
EP02_exception_list_recall	0.825	+0.437	0.825	+0.449
SM01_preference_adoption	0.734	+0.134	0.600	+0.012
SM02_constraint_retention	0.885	+0.191	0.870	+0.188
SM05_weak_trigger_preference_adoption	0.823	+0.223	0.955	+0.367
Procedural
PC01_sop_bootstrap_01	0.497	+0.136	0.520	-0.079
PC01_sop_bootstrap_02	0.594	-0.124	0.333	-0.115
PC01_sop_bootstrap_03	0.873	+0.143	0.478	+0.026
PC01_sop_bootstrap_04	0.485	-0.227	0.578	+0.027
PC01_sop_bootstrap_05	0.488	+0.117	0.231	+0.022
PC01_sop_bootstrap_06	0.488	+0.117	0.231	-0.005
PC03_latent_rule_induction_01	0.493	+0.126	0.317	-0.031
PC04_failure_to_rule_01	0.482	+0.111	0.353	-0.005
Information Gathering
PG01_release_decision_followup	0.647	+0.267	0.710	+0.121
PG02_ops_exception_desk	0.818	-0.059	0.748	+0.164
PG03_oncall_handoff_lookup	0.790	+0.425	0.730	+0.116
PG04_temporary_waiver_audit	0.755	+0.097	0.724	+0.169
PG05_change_freeze_followup	0.549	-0.065	0.699	+0.122
PG06_kappa_integration_review	0.701	-0.124	0.769	+0.029
Update
EP03_recall_then_modify	0.762	+0.347	0.631	+0.190
PC02_sop_patch_01	0.506	+0.041	0.659	+0.163
PC02_sop_patch_02	0.663	+0.017	0.554	-0.127
SM03_fact_correction	0.518	-0.078	0.965	+0.492
SM04_rule_migration	0.492	+0.135	0.589	+0.226
SM06_temporary_exception_pollution	0.697	+0.034	0.938	+0.235
SM07_scoped_rule_migration	0.701	+0.354	0.815	+0.502
Overall	0.66	+0.13	0.66	+0.15

D.5 Score Variance Across Runs

Two configurations have three independent runs: Hermes/MiniMax-M2.7 and Hermes+/MiniMax-M2.7. Table 11 reports the mean and standard deviation of the w/ evolve score and across runs, aggregated by capability.

媒体内容 · 前往原文查看

	Hermes				Hermes+
Capability
Memory
Procedural						0.04		0.04
Info Gathering
Update
Overall

D.6 Computational Cost

Table 12 reports the wall-clock time and token usage for the reported model–framework configurations, as observed during our evaluation runs.

媒体内容 · 前往原文查看

Framework	Model	Tokens/ep	Wall-time/ep (s)
Hermes	GLM-5.1	16,928	116.9
	Kimi K2.6	19,581	257.9
	MiniMax-M2.7	12,615	70.5
	DeepSeek-V4-Pro	12,473	134.9
	Claude Opus 4.6	19,986	27.1
	Claude Sonnet 4.6	5,551	27.5
	GPT-5.4	8,964	89.9
Hermes+	MiniMax-M2.7	31,859	77.4
	DeepSeek-V4-Pro	11,218	46.0
	GPT-5.4	13,504	114.1
	Claude Sonnet 4.6	10,203	48.5
	Claude Opus 4.6	11,022	44.7
nanobot	MiniMax-M2.7	10,905	66.7
ZeroClaw	MiniMax-M2.7	43,955	65.3
Agent-Zero	MiniMax-M2.7	53,100	117.5

Hermes vs. Hermes+ cost.

Table 12 shows Hermes+ uses approximately 2.5 more tokens per episode than Base Hermes (31,859 vs. 12,615). The increase comes from the planning prompt (E1), structured memory rendering (E2), skill-list queries (E3), gating retries (E4), and closeout review (E5). Wall-clock time increases by only 1.10 (77.4 s vs. 70.5 s), since most additional tokens are added to the system-prompt context rather than to generated output. Kimi K2.6 shows the highest wall-time per episode (257.9 s) due to higher API latency.

Appendix E Reproducibility Details

E.1 Context and Persistence Handling

Every episode starts in a fresh session. The benchmark does not append dialogue from earlier episodes. Persistence-on exposes state from earlier episodes through the agent’s native memory, skill, or history interface. Persistence-off removes access to that state.

Table 13 separates the settings held fixed from native system differences. Within each matched pair, the model, agent, prompt, tools, context window, output limits, agent limits, and compaction policy are fixed. Only access to retained state changes. The benchmark does not add a shared truncation rule. As a result, controls for context policy within a model–agent pair, while absolute scores across systems still include native context-management differences.

媒体内容 · 前往原文查看

Comparison	Held fixed	Native difference retained
Model comparison	Hermes, tasks, graders, and benchmark limits	Context window and provider context policy
Agent comparison	MiniMax-M2.7, tasks, and graders	Memory rendering, compaction, and truncation policy
Persistence-on/off pair	Model, agent, prompt, tools, context window, output limits, agent limits, and compaction policy	Access to retained state

E.2 Model Inference Settings

Table 14 reports model-side settings. Values without an asterisk are sent by our code. An asterisk marks a provider-documented default used when the request does not set that field. A dash means the request does not set the field and no documented default was found for that model and API route.

媒体内容 · 前往原文查看

Model and API route	Reasoning	Temp.	Top-	Max output	Agent limit
GPT-5.4, OpenAI Responses	effort=medium			–	50
Claude Opus 4.6, OpenRouter Chat Completions	enabled, effort=medium	0		–	50
Claude Sonnet 4.6, Anthropic Messages	off∗	0	–	16,384	50
DeepSeek-V4-Pro, Anthropic-compatible Messages	enabled∗, effort=high∗	0	–	16,384	50
GLM-5.1, Anthropic-compatible Messages	enabled∗	0		16,384	50
Kimi K2.6, OpenAI-compatible Chat Completions	enabled∗	0		16,384	50
MiniMax-M2.7, Anthropic-compatible Messages	enabled∗	0		16,384	50

The open-ended LLM judge uses MiniMax-M2.7 with temperature 0 and a maximum output of 8,192 tokens.

E.3 Agent Limits and Retry Policies

The fixed-model comparison keeps MiniMax-M2.7 constant but preserves each agent’s native loop. Table 15 reports the stopping and retry rules. The task files set an outer limit of 25 turns. Each adapter runs its native loop inside one benchmark step. Persistence-on and persistence-off always use the same settings.

媒体内容 · 前往原文查看

Agent	Stop condition and limit	Model-call retry policy
Hermes / Hermes+	Final response, 50 iterations, or 300-second timeout	Up to 3 attempts with exponential backoff; streaming layer retries up to 2 times
nanobot	Final response, 30 iterations, or 300-second timeout	3 retries with 1, 2, and 4 second delays
ZeroClaw	No more tool calls, recursion limit 100, or 300-second timeout	No retry added by the benchmark adapter
Agent-Zero	response tool or 1,200-second timeout	Up to 2 retries with a 1.5 second delay

A timed-out or crashed episode receives a score of zero, and the benchmark continues to the next episode. The failed episode is not rerun.

Appendix F Extended Related Work

This appendix records additional distinctions between trajectory diagnosis and persistence mechanisms.

Interactive and trajectory-level evaluation.

AgentBoard (ma2024agentboard) introduces fine-grained progress metrics, TRAJECT-Bench (he2025traject) scores tool-call sequences along exact match, inclusion, parameter usage, and LLM-judge satisfaction, and ATBench (li2026atbench) grades multi-turn safety traces under delayed-trigger protocols. These methods provide detailed evidence about actions produced within a task; PAST-Bench uses such evidence to diagnose whether retained state is reused across later episodes of the same family. This distinction is temporal as well as diagnostic: a within-task judge can identify whether an action was useful or unsafe, but cannot establish that an artifact written in one episode caused success in a later fresh session. Conversely, an endpoint comparison across sessions can show improvement while leaving the responsible persistence channel ambiguous. PAST-Bench combines matched later outcomes with trace evidence so that these two questions remain separate.

Memory, procedural, and architectural mechanisms.

LongMemEval (wu2025longmemeval) and LoCoMo (maharana2024evaluating) stress information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. SkillsBench (li2026skillsbench) and work on skill optimization, curation, and lifecycle management (yang2026skillopt; ouyang2026skillos; huang2026rawexperience; lin2026museautoskill) study reusable procedures; direct corpus interaction (li2026beyond) broadens agentic retrieval beyond fixed similarity interfaces, while AgentArch (bogavelli2025agentarch) compares orchestration, prompting, memory, and tool choices. These component-level analyses complement family-level tests of whether retained state improves later executable tasks. They also hold different objects fixed: memory evaluations typically retain one memory interface, skill studies intervene on reusable artifacts, and architecture studies compare bundled design choices. PAST-Bench instead fixes the framework for model comparisons and the model for framework comparisons, then toggles access to retained state within matched task families. It therefore tests a narrower causal question and does not replace substrate-specific measures of memory, skill, or architecture quality.