1 Introduction

Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents for sequential decision-making Chen et al. (2023b); Fu et al. (2025); Jin et al. (2025); Mei et al. (2025). Experiential memory further enables these agents to reuse prior successes and avoid repeated mistakes Wang et al. (2024); Yan et al. (2025); Xu et al. (2026). However, most memory-augmented agents follow a verbatim replay paradigm: retrieved trajectories or principles are treated as static records and directly inserted into the model context Yang et al. (2026); Zhang et al. (2025a). This design conflates retrieval relevance with action-level applicability. A memory may be semantically relevant to the task yet inappropriate for the current interaction state, because it was formed under different environmental conditions. An alternative line of work internalizes experience into model parameters, enabling state-conditioned behavior without explicitly replaying retrieved records Zhang et al. (2025b). While such parametric memory offers adaptive generation, the underlying experience and its influence on a decision remain implicit and difficult to inspect or revise. Explicit memory banks provide greater traceability, but typically lack this adaptive use of experience. This exposes a desirable middle ground: retaining explicit, attributable memories while dynamically reconstructing them for the current state.

This middle ground echoes a fundamental insight from cognitive science: human remembering is reconstructive rather than reproductive Loftus and Palmer (1974). Recall does not recover an immutable record of the past; instead, retrieved experience is interpreted and reorganized using present cues and prior knowledge before it informs behavior. As illustrated in Figure 1, prior agents largely connect retrieval directly to action generation, whereas human memory introduces an intermediate process that evaluates and adapts recalled experience. This suggests a broader paradigm shift for memory-augmented agents—from retrieve and replay to retrieve, evaluate, and reconstruct.

Inspired by this mechanism, we decompose memory-guided decision-making into five stages that parallel the reconstructive process: environment observation captures the current situation; experience retrieval recalls a potentially relevant memory together with its historical context; memory critique assesses its applicability and identifies state mismatches; contextual memory reconstruction preserves transferable knowledge while revising or discarding incompatible content; and action generation uses the reconstructed guidance to make a context-aligned decision.

Based on this formulation, we introduce MemHarness, designed to actively harness retrieved experiences by explicitly inserting critique and reconstruction between retrieval and action, turning memory from a static prompt fragment into context-sensitive guidance.

Building on this formulation, MemHarness bridges explicit and parametric memory utilization: it retains retrieved experiences as inspectable records, while parameterizing their state-conditioned critique and reconstruction within a single policy. At each decision step, conditioned on the current observation, agent first critiques the retrieved experience against its original context, then reconstructs it into state-specific guidance, and finally generates the executable action. Crucially, this reconstructive ability requires no additional human annotation: it emerges through end-to-end training with Group Relative Policy Optimization (GRPO). By optimizing for task success, the agent implicitly learns a discriminative reconstruction strategy, preserving transferable memories while rewriting misleading ones to close the gap between historical knowledge and the present state.

We evaluate MemHarness on two challenging agent decision-making benchmarks, ALFWorld and WebShop. It consistently outperforms both pure RL baselines and state-of-the-art static memory-augmented methods. Ablation studies further show that removing the reconstruction stage degrades performance to the level of naive memory replay, confirming that adaptive reconstruction, rather than retrieval alone, is the primary driver of the observed gains.

• •

  A reconstructive view of agent memory. We identify the applicability gap underlying the prevalent verbatim replay paradigm, and, drawing on the reconstructive nature of human memory, recast memory-guided decision-making as a five-stage process spanning observation, retrieval, critique, reconstruction, and action.

• •

  A reconstruction-centric framework. We propose MemHarness, which actively harnesses retrieved experiences by inserting explicit memory reconstruction between retrieval and action, and learns this capability end-to-end via GRPO without external supervision, combining the traceability of explicit memory with the adaptivity of parameterized reconstruction.

• •

  Comprehensive empirical validation. On interactive decision-making benchmarks, MemHarness delivers robust improvements over static replay, with ablations isolating reconstruction as the key contributing factor.

2 Related Works

2.1 LLM Agents for Interactive Decision-Making

LLM agents navigate interactive tasks using prompting paradigms like ReAct Yao et al. (2023); Wei et al. (2022), often augmented with planning or self-reflection Shinn et al. (2023); Ma et al. (2026). However, these training-free methods are bounded by frozen model capabilities. To acquire task-specific skills from interaction, agents are fine-tuned via supervised learning Chen et al. (2024b, 2023a) or reinforcement learning (RL) Schulman et al. (2017); Ahmadian et al. (2024); Shao et al. (2024); Feng et al. (2026). While RL optimizes decision-making directly, standard frameworks lack explicit mechanisms to accumulate and reuse cross-episode experiences, motivating memory-augmented agents.

2.2 Memory-Augmented LLM Agents

Memory-augmented agents address this by retrieving past interactions or distilled insights Zhao et al. (2024); Fang et al. (2026); Chhikara et al. (2025); Liu et al. (2026). Most rely on frozen policies, while alternative approaches internalize experience directly into model weights Zhang et al. (2025b), sacrificing the traceability of explicit memory. Recent works integrate external memory with RL Zhang et al. (2026); Wu et al. (2025), yet they inject retrieved experiences verbatim. Because interactive tasks exhibit high state variance, replay often introduces misaligned or misleading guidance.

MemHarness bridges these two paradigms. It retains explicit memory banks for traceability but parameterizes their state-conditioned reconstruction. Rather than applying retrieved memory blindly, MemHarness is optimized end-to-end to critique and rewrite past experiences into context-specific guidance before acting.

3 Method

3.1 Problem Formulation

Sequential Decision-Making.

We model interactive decision-making as a POMDP , where , , and are the state, action, and observation spaces, respectively, and and denote the transition and reward functions. Given a task description , at step the agent observes and selects using a policy . To account for partial observability, the policy conditions on a recent history window . The objective is to maximize the expected episodic return .

Experiential Memory.

The agent maintains a memory bank by periodically distilling collected trajectories into natural-language experiences. Each entry contains an abstract strategy and its source observation . Following Wu et al. (2025), we perform semantic deduplication, track empirical utility and periodically prune low-utility entries. Given a context-derived query , a retriever returns the top- entries:

(1)

The source observations enable comparison between historical and current states during reconstruction.

From Replay to Reconstruction.

Verbatim replay directly conditions action generation on retrieved experiences: . MemHarness instead first reconstructs them into state-specific guidance and then acts:

				(2)

where maps the raw reconstruction to final guidance , including rejection of mismatched memories. Both stages share the unified policy .

3.2 Overall Framework

We introduce MemHarness, an RL-driven framework that reconstructs retrieved experiences for the current state rather than replaying them verbatim. As shown in Figure 2, its inference pipeline comprises three stages: retrieval, contextual memory reconstruction, and action generation. These three stages instantiate the five-stage formulation in Section 1: the history window serves as the environment observation, and memory critique is folded into the reconstruction stage.

1. 1.

   Retrieval. At step , the agent decides whether to query the experiential memory bank based on . If triggered, retrieval returns relevant experiences , which are reconstructed rather than directly added to the action context.

2. 2.

   Reconstruction. The policy critiques against and generates state-specific guidance , retaining applicable knowledge while discarding or revising misaligned content.

3. 3.

   Action Generation. The same policy generates an executable action conditioned on and .

MemHarness thus separates reconstruction from action generation while implementing both with a unified policy. Because supervised reconstruction traces are unavailable, we train the two-step process end-to-end with GRPO using only task rewards. The following sections detail the reconstruction and optimization procedures.

3.3 Contextual Memory Reconstruction

MemHarness transforms retrieved experiences into state-specific guidance through an explicit reconstruction. Because experiences abstracted from historical trajectories may be outdated or incompatible with the current state, the policy critiques their applicability before they influence action generation.

Critique and Reconstruction.

The reconstruction context concatenates the task description , current history , and retrieved memory tuples :

(3)

where is a retrieved experience, is its source observation, and denotes text concatenation. The policy then generates:

(4)

By comparing each source observation with the current context represented by , the policy identifies state shifts and retains, revises or rejects the retrieved experience to produce context-specific guidance.

If no retrieved experience is applicable, the policy outputs <EMPTY>. The guidance used for action generation is

(5)

where instructs the agent to rely on its own reasoning without memory guidance.

Action Generation.

The same policy generates the executable action conditioned on the reconstructed guidance:

(6)

Thus, action generation uses either adapted memory or independent reasoning, rather than unfiltered historical records. Reconstruction and action generation share parameters and are optimized jointly. Since has no ground-truth annotation, we treat it as a latent reasoning process and train the pipeline end-to-end using task-level rewards

3.4 Training with GRPO

Since the guidance is a latent variable without ground-truth supervision, we optimize the retrieve–reconstruct–act pipeline end-to-end using reinforcement learning. The reward combines task outcomes with format-based signals that enforce valid reconstruction and action outputs. We adopt GRPO to estimate advantages from groups of sampled trajectories without training a separate value network.

Reward Design.

The reward for a trajectory primarily depends on the sparse task outcome (0 for failure, 10 for success), augmented by a minor bonus to enforce structural constraints: . The format score equally weights three criteria: (1) exactly one valid <think> and <action> block per step, (2) appropriate memory retrieval frequency with <retrieve_memory> block (1 to 5 times per episode), and (3) strict English output. This formatting bonus explicitly encourages an agentic protocol (i.e., reason first, retrieve memory when necessary, single action per step) without overshadowing the primary task objective.

Policy Update.

For each task, we sample a group of trajectories . GRPO computes the group-normalized advantage:

(7)

The trajectory-level advantage is assigned to all tokens in , providing shared credit to thinking process, reconstruction and action generation. For token , the importance ratio is

(8)

The clipped surrogate objective is

(9)

and the overall objective is

(10)

where controls clipping and weights the KL regularization. This objective jointly optimizes memory reconstruction and action generation using the same trajectory-level reward.

4 Experiments

In this section, we empirically evaluate our proposed MemHarness framework on long-horizon interactive environments. Our evaluation is designed to answer the following core research questions (RQs):

• •

  RQ1: Does the MemHarness framework outperform existing agentic and memory-augmented baselines on complex interactive tasks? (Section 4.2)

• •

  RQ2: How do individual components (especially the contextual memory reconstruction module) contribute to the overall performance? (Section 4.3)

• •

  RQ3: Is MemHarness robust against environmental distribution shifts in out-of-distribution (OOD) scenarios? (Section 4.4)

• •

  RQ4: Does the reconstruction mechanism genuinely rely on state comparison to make decisions? (Section 4.5)

• •

  RQ5: How do adaptive memory utilization behaviors evolve during reinforcement learning, and how do they correlate with task success? (Section 4.6)

4.1 Experimental Setup

Benchmarks and Baselines.

We evaluate MemHarness on two complex, long-horizon interactive benchmarks: ALFWorld Shridhar et al. (2020) (embodied household tasks) and WebShop Yao et al. (2022) (goal-directed online shopping). Performance is measured by task success rate, together with the average task score on WebShop. We compare MemHarness against closed-source LLMs, including GPT-4o Hurst et al. (2024) and Gemini-2.5-Pro Comanici et al. (2025), prompt-based or memory-based agents (ReAct, Reflexion, Mem0, ExpeL, MemP, and SimpleMem) Yao et al. (2023); Shinn et al. (2023); Chhikara et al. (2025); Zhao et al. (2024); Fang et al. (2026); Liu et al. (2026), and RL-trained agents (RLOO, GRPO, MemRL, EvolveR, Mem0+GRPO, and SimpleMem+GRPO) Ahmadian et al. (2024); Guo et al. (2025); Zhang et al. (2026); Wu et al. (2025); Liu et al. (2026). We also evaluate ablated variants with verbatim memory replay or the reconstruction harness removed at inference time to isolate the contribution of our design.

Implementation Details.

We adopt Qwen2.5-7B-Instruct Qwen et al. (2025) as the policy backbone. Our implementation builds on the verl-agent framework Feng et al. (2026). Following Deepseek-R1 and EvolveR Guo et al. (2025); Wu et al. (2025), we perform a brief cold-start stage before RL training. For each benchmark, the cold-start dataset contains 200 multi-turn interaction trajectories with chain-of-thought reasoning and active retrieval of memories, as well as 200 question–answer examples that summarize trajectories into memories. Whenever memory is retrieved, the agent is required to reconstruct it before taking an action. This stage primarily aligns the model with the required interaction and memory-abstraction formats. The unified policy is subsequently optimized end-to-end via GRPO with sparse task rewards ( for success and for failure) and a minor formatting bonus. For fair comparison, all trainable baselines use the same backbone, observation history window (), interaction budget, and optimization configuration. We will show more details in the Appendix.

Memory Configuration.

The experiential memory bank is initialized as empty and progressively populated during RL training with memory principles summarized by the policy from its own interaction trajectories. We use BGE-M3 Chen et al. (2024a) as the embedding model and retrieve the top- most relevant principles.

媒体内容 · 前往原文查看

Method	AlfWorld							WebShop
	Pick	Look	Clean	Heat	Cool	Pick2	Avg. SR	Score	SR
Closed-source LLMs
GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	49.2	31.8	23.7
Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	62.1	42.5	35.9
Qwen2.5-7B-Instruct
Qwen2.5	33.4	21.6	19.3	6.9	2.8	3.2	14.5	26.4	7.8
Prompt-based Agentic or Memory-based Methods
ReAct	48.5	35.4	34.3	13.2	18.2	17.6	27.9	46.2	19.5
Reflexion	62.0	41.6	44.9	30.9	36.3	23.8	39.9	58.1	28.8
Mem0	54.0	55.0	26.9	36.4	20.8	7.7	33.5	23.9	2.0
ExpeL	21.0	67.0	55.0	52.0	71.0	6.0	45.3	30.9	11.2
MemP	54.3	38.5	48.1	56.2	32.0	16.7	41.0	25.3	6.4
SimpleMem	64.5	33.3	20.0	12.5	33.3	3.8	27.9	33.2	8.6
RL-based Methods
RLOO	87.6	78.2	87.3	81.3	71.9	48.9	75.9	80.3	65.7
GRPO	90.8	66.1	89.3	74.7	72.5	64.7	76.4	79.3	66.1
Memory-Augmented RL-based Methods
MemRL	62.8	38.5	22.2	12.5	8.0	0.0	24.0	29.5	9.2
Mem0+GRPO	78.1	54.8	56.1	31.0	65.0	26.9	52.0	58.1	37.5
SimpleMem+GRPO	89.5	36.3	60.0	50.0	64.9	26.3	54.5	67.8	46.9
EvolveR (reproduced)	84.8	61.5	95.7	62.5	61.9	54.2	70.1	84.1	72.6
MemHarness	87.0	78.6	97.0	87.5	71.4	90.0	85.2	87.4	75.6

4.2 Main Results: Overall Effectiveness

Table 1 summarizes the performance on AlfWorld and WebShop. Overall, MemHarness achieves the best results, reaching success rates of 85.2% on AlfWorld and 75.6% on WebShop.

Comparison with Foundation Models. MemHarness demonstrates massive improvements over its base model, Qwen2.5-7B-Instruct, and substantially outperforms strong closed-source models like Gemini-2.5-Pro (+23.1% on AlfWorld and +39.7% on WebShop) despite its 7B parameter scale.

Advantage over Existing Paradigms. Prompt-based and test-time memory methods (e.g., Reflexion, ExpeL) yield limited success without policy optimization. While pure RL (GRPO) provides a strong baseline, naively combining it with external memory (e.g., Mem0+GRPO, SimpleMem+GRPO) severely degrades performance. MemHarness effectively overcomes this issue via state-conditioned memory reconstruction, safely leveraging historical experience to outperform standard GRPO by absolute margins of +8.8% and +9.5% on AlfWorld and WebShop, respectively.

4.3 Ablation Studies: Component Analysis

To evaluate the contribution of each component (RQ2), we conduct ablation studies on ALFWorld and WebShop, with results summarized in Table 2. We identify three key findings regarding memory utilization and policy optimization:

Necessity of RL.

The cold-start model, trained solely for format alignment, performs poorly (7.6% on ALFWorld), confirming that RL is necessary for acquiring actual interactive decision-making capabilities.

Raw Memory Introduces Noise.

Injecting unrefined memory (“RL + Raw Memory” or “w/o reconstruction”) hurts ALFWorld performance. This proves verbatim replay introduces state-mismatch noise, which our reconstruction mechanism effectively mitigates.

Intrinsic Policy Improvement.

Disabling test-time memory (“w/o memory”) still substantially outperforms the ‘RL Only‘ baseline (83.0% vs. 76.4% on ALFWorld). Thus, the reconstruction objective during training intrinsically enhances the agent’s base reasoning capabilities, serving as high-quality latent guidance.

Necessity of Policy-Internal Reconstruction.

Replacing MemHarness’s internal reconstruction module with a generic instruction-tuned LLM (Qwen2.5-7B-Instruct) while keeping the actor unchanged degrades performance significantly (e.g., 85.2% to 77.7% on ALFWorld). This demonstrates that zero-shot text rewriting is insufficient; end-to-end RL is crucial for learning task-grounded adaptation that aligns with environment dynamics.

媒体内容 · 前往原文查看

Method	Test-time		ALFWorld							WebShop
	Memory	Reconstruction	Pick	Look	Clean	Heat	Cool	Pick2	Avg. SR	Avg. SR
Base Model	✗	✗	33.4	21.6	19.3	6.9	2.8	3.2	14.5	7.8
Cold Start Model	✗	✗	11.8	21.1	0	7.1	0	5.3	7.6	17.6
RL Only (GRPO)	✗	✗	90.8	66.1	89.3	74.7	72.5	64.7	76.4	66.1
RL + Raw Memory	✓	✗	84.8	61.5	95.7	62.5	61.9	54.2	70.1	72.6
MemHarness (Ours)	✓	✓	87.0	78.6	97.0	87.5	71.4	90	85.2	75.6
– generic LLM reconstrucion	✓	✓	97.1	52.6	79.2	78.6	80	78.9	77.7	71.8
– w/o reconstruction	✓	✗	94.1	52.6	79.2	85.7	76.7	89.5	79.6	74.6
– w/o memory	✗	✗	100	68.4	91.7	78.6	70	89.5	83.0	73.6

4.4 Out-of-Distribution Generalization

To address RQ3, we evaluate robustness against environmental shifts on ALFWorld Out-of-Distribution (OOD) settings, where room layouts and object placements are unseen during training. In such scenarios, directly applying historical memories risks severe state mismatch.

As shown in Table 3, MemHarness achieves the highest average success rate (85.9%), substantially outperforming the “RL + Raw Memory” baseline (76.3%). Notably, while the intrinsic policy (“w/o memory”) remains competitive (83.0%), injecting unadapted memories (“w/o reconstruction”) degrades performance to 82.4%. This confirms that verbatim memory replay in unseen environments introduces noise. MemHarness overcomes this by dynamically filtering and rewriting mismatched guidance, safely leveraging past experiences to achieve superior OOD generalization.

媒体内容 · 前往原文查看

Method	ALFWorld OOD Success Rate (%)
	Pick	Look	Clean	Heat	Cool	Pick2	Avg. SR
MemHarness	97.1	73.7	87.5	92.9	80.0	84.2	85.9
– w/o reconstruction	94.1	52.6	87.5	92.9	83.3	84.2	82.4
– w/o memory	100.0	68.4	91.7	78.6	70.0	89.5	83.0
RL + Raw Memory	91.2	68.4	75.0	78.6	86.7	57.9	76.3

4.5 Mechanism Deep Dive: How is Memory Reconstructed?

To answer RQ4, we investigate whether reconstruction genuinely relies on state comparison, examining its behavior from macro-level filtering to micro-level adaptation.

Macro-level Filtering.

We ablate the historical source state () provided during reconstruction (Table 4). Removing entirely leaves the rejection rate unchanged but degrades task success (e.g., on ALFWorld), as the agent accepts misaligned guidance. Crucially, replacing with a mismatched state from a random memory markedly increases rejection rates ( on ALFWorld, on WebShop). This confirms the policy actively compares historical and current states to filter incompatible memories, rather than reacting to missing fields, thereby maintaining robust downstream performance.

Micro-level Counterfactual Adaptation.

We construct 1,000 offline probes by asking a strong LLM to minimally edit each current state so that its retrieved memory becomes inapplicable, and evaluate only the reconstruction output rather than task execution (Appendix B). Table 4 reports the resulting within-environment shifts. In ALFWorld, counterfactual edits reduce unchanged outputs and increase adaptation and rejection. In WebShop, they further increase rejection. The different response patterns are consistent with their observation formats: ALFWorld provides concise state descriptions, whereas WebShop exposes substantially longer webpage content. Despite this difference, both show that reconstruction responds to fine-grained state changes.

媒体内容 · 前往原文查看

(a) Source-state ablation

Source State	ALFWorld		WebShop
	RR (%)	SR (%)	RR (%)	SR (%)
Correct source ()	8.7	85.2	56.0	75.6
No source (w/o )	7.8	80.0	55.9	73.0
Random source	13.3	84.3	63.3	73.6

(b) Counterfactual behavior

Task	State	Output Distribution (%)
		Unchanged	Adapted	Reject
ALFWorld	Match ()	46.0	53.4	0.6
	Edit ()	37.3	56.3	6.4
WebShop	Match ()	0.0	27.9	72.1
	Edit ()	0.0	21.2	78.8

4.6 Training Dynamics and Behavioral Evolution (RQ5)

To address RQ5, we track internal memory decisions and task success rates during RL training. MemHarness learns task-specific memory strategies rather than fixed rules (Figure 3). ALFWorld converges to a sparse policy (retrievals dropping to 2–3 per trajectory), whereas WebShop maintains frequent retrieval with persistent active filtering. These learned decisions align directly with task outcomes (Figure 4). Across both environments, trajectories containing accepted reconstructions consistently achieve higher success rates than those with rejected ones, confirming the agent effectively learns to judge memory applicability.

5 Conclusion

We introduced MemHarness, an RL-driven framework that shifts memory-augmented agents from verbatim replay to state-conditioned reconstruction. Optimized end-to-end via GRPO, a single unified policy learns to autonomously critique and rewrite historical experiences into state-specific guidance before acting. Experiments on ALFWorld and WebShop show our 7B model significantly outperforms closed-source models and RL baselines, especially in out-of-distribution environments. Analyses reveal this mechanism actively filters mismatched noise and fundamentally improves the agent’s intrinsic reasoning, even when memory is unavailable at inference. Future work will explore scaling to larger models and open-ended environments.

References

• [1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker (2024-08) Back to basics: revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 12248–12267. External Links: Link, Document Cited by: §2.1, §4.1.
• [2] B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao (2023) Fireact: toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. Cited by: §2.1.
• [3] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu (2024) Bge m3-embedding: multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216. Cited by: §A.3, §4.1.
• [4] L. Chen, L. Wang, H. Dong, Y. Du, J. Yan, F. Yang, S. Li, P. Zhao, S. Qin, S. Rajmohan, et al. (2023) Introspective tips: large language model for in-context decision making. arXiv preprint arXiv:2305.11598. Cited by: §1.
• [5] Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao (2024) Agent-flan: designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 9354–9366. Cited by: §2.1.
• [6] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Cited by: §2.2, §4.1.
• [7] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §4.1.
• [8] R. Fang, Y. Liang, X. Wang, J. Wu, S. Qiao, P. Xie, F. Huang, H. Chen, and N. Zhang (2026) Memp: exploring agent procedural memory. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 17490–17502. Cited by: §2.2, §4.1.
• [9] L. Feng, Z. Xue, T. Liu, and B. An (2026) Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems 38, pp. 46375–46408. Cited by: §2.1, §4.1.
• [10] D. Fu, J. Mei, L. Wen, X. Yang, C. Yang, R. Wu, T. Hu, S. Li, Y. Shen, X. Cai, et al. (2025) RE-searcher: robust agentic search with goal-oriented planning and self-reflection. arXiv preprint arXiv:2509.26048. Cited by: §1.
• [11] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §4.1, §4.1.
• [12] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §4.1.
• [13] B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. External Links: 2503.09516, Link Cited by: §1.
• [14] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: 4th item.
• [15] J. Liu, Y. Su, P. Xia, S. Han, Z. Zheng, C. Xie, M. Ding, and H. Yao (2026) SimpleMem: efficient lifelong memory for llm agents. arXiv preprint arXiv:2601.02553. Cited by: §2.2, §4.1.
• [16] E. F. Loftus and J. C. Palmer (1974) Reconstruction of automobile destruction: an example of the interaction between language and memory. Journal of verbal learning and verbal behavior 13 (5), pp. 585–589. Cited by: §1.
• [17] E. Ma, L. Zhou, T. Tang, J. Zhang, J. Jiang, Z. Zhang, D. Han, K. Zhan, X. Zhang, X. Lang, et al. (2026) Correctad: a self-correcting agentic system to improve end-to-end planning in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, pp. 7755–7763. Cited by: §2.1.
• [18] J. Mei, T. Hu, D. Fu, L. Wen, X. Yang, R. Wu, P. Cai, X. Cai, X. Gao, Y. Yang, et al. (2025) -Searcher: a searching-based agent model for open-domain open-ended question answering. arXiv preprint arXiv:2505.16582. Cited by: §1.
• [19] Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2025) Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §A.2, §4.1.
• [20] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.1.
• [21] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §A.2, §2.1.
• [22] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2024) HybridFlow: a flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256. Cited by: §A.2.
• [23] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. External Links: 2303.11366, Link Cited by: §2.1, §4.1.
• [24] M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2020) Alfworld: aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768. Cited by: §A.1, §4.1.
• [25] J. Wang, X. Yi, R. Guo, H. Jin, P. Xu, S. Li, X. Wang, X. Guo, C. Li, X. Xu, et al. (2021) Milvus: a purpose-built vector data management system. In Proceedings of the 2021 International Conference on Management of Data, pp. 2614–2627. Cited by: §A.3.
• [26] L. Wang, X. Zhang, H. Su, and J. Zhu (2024) A comprehensive survey of continual learning: theory, method and application. External Links: 2302.00487, Link Cited by: §1.
• [27] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, pp. 24824–24837. Cited by: §2.1.
• [28] R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang, et al. (2025) Evolver: self-evolving llm agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079. Cited by: §A.3, §2.2, §3.1, §4.1, §4.1.
• [29] Z. Xi, Y. Ding, W. Chen, B. Hong, H. Guo, J. Wang, D. Yang, C. Liao, X. Guo, W. He, S. Gao, L. Chen, R. Zheng, Y. Zou, T. Gui, Q. Zhang, X. Qiu, X. Huang, Z. Wu, and Y. Jiang (2024) AgentGym: evolving large language model-based agents across diverse environments. External Links: 2406.04151 Cited by: §A.4.
• [30] W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2026) A-mem: agentic memory for llm agents. Advances in Neural Information Processing Systems 38, pp. 17577–17604. Cited by: §1.
• [31] S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, H. Schütze, V. Tresp, and Y. Ma (2025) Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv preprint arXiv:2508.19828. Cited by: §1.
• [32] C. Yang, X. Yang, L. Wen, D. Fu, J. Mei, R. Wu, P. Cai, Y. Shen, N. Deng, J. Xu, et al. (2026) Towards self-evolving agents: enabling autonomy through interactive experience refinement. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 30424–30451. Cited by: §1.
• [33] S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022) WebShop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 20744–20757. External Links: Link Cited by: §A.1, §4.1.
• [34] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. External Links: 2210.03629, Link Cited by: §2.1, §4.1.
• [35] G. Zhang, M. Fu, G. Wan, M. Yu, K. Wang, and S. Yan (2025) G-memory: tracing hierarchical memory for multi-agent systems. External Links: 2506.07398, Link Cited by: §1.
• [36] G. Zhang, M. Fu, and S. Yan (2025) Memgen: weaving generative latent memory for self-evolving agents. arXiv preprint arXiv:2509.24704. Cited by: §1, §2.2.
• [37] S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li, et al. (2026) Memrl: self-evolving agents via runtime reinforcement learning on episodic memory. arXiv preprint arXiv:2601.03192. Cited by: §2.2, §4.1.
• [38] A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang (2024) ExpeL: llm agents are experiential learners. External Links: 2308.10144, Link Cited by: §2.2, §4.1.

Appendix A Implementation Details

A.1 Environment and Evaluation Setup

We evaluate our method on two challenging interactive benchmarks: ALFWorld and WebShop. ALFWorld Shridhar et al. (2020) is an embodied text-based environment containing six sub-task categories (e.g., Pick, Look, Clean). The maximum interaction horizon is set to 50 steps, with a maximum prompt length of 2048 tokens and a generation limit of 512 tokens. WebShop Yao et al. (2022) is an e-commerce website simulation containing over 1.1 million products. The maximum interaction horizon is restricted to 15 steps, with a maximum prompt length of 4096 tokens and a generation limit of 512 tokens.

Reward Function Design: Across both environments, we employ a straightforward outcome-driven reward function combined with a format penalty. The agent receives a sparse outcome reward of upon successful task completion and for failure. To encourage adherence to the prescribed interaction protocol, we add a small formatting bonus, weighted by , to the sparse outcome reward. The format score equally weights three criteria: (1) each step contains exactly one valid block followed by one valid <action>…</action> block, (2) memory is retrieved through valid <retrieve_memory>…</retrieve_memory> blocks between one and five times per episode, (3) thinking contents are including by one valid <think>…</think> block and (4) all generated content is strictly in English.

A.2 Training Hyperparameters

Our reinforcement learning optimization is based on Group Relative Policy Optimization (GRPO) Shao et al. (2024) and implemented using the veRL Sheng et al. (2024) framework. We initialize the policy with instruction-tuned base models Qwen2.5-7B Qwen et al. (2025) and freeze the reference policy during training to compute the KL divergence penalty.

The core hyperparameter settings are consistent across both ALFWorld and WebShop:

• •

  Rollout & Sampling: During the environment interaction phase, we sample a group size of trajectories for each prompt. The rollout generation temperature is set to , and the validation temperatue is set to . We sample 16 different groups per rollout, utilizing a total of 128 parallel environments.

• •

  Optimization: The policy network is updated using the Adam optimizer with a learning rate of .

• •

  GRPO Objectives: The PPO-style clipping parameter is set to . To prevent the policy from degrading and deviating too far from the base model, the KL divergence penalty coefficient is set to .

• •

  Batching: The training utilizes a mini-batch size of 256 for Alfworld and 64 for Webshop with 1 tensor parallel size. We use vLLM Kwon et al. (2023) for high-throughput rollout generation with a GPU memory utilization ratio of 0.75.

A.3 Memory Bank and Retrieval Setup

Following prior work, we implement the memory store as a vector database using Milvus Wang et al. (2021) for efficient similarity search. Each memory entry consists of a distilled experience paired with its source interaction state . We use BGE-M3 Chen et al. (2024a) as the embedding model to encode both the retrieval query and the stored entries, and retrieve the top- () most relevant experiences at each step based on cosine similarity.

Memory Initialization. Inspired by prior work Wu et al. (2025), the memory bank is not manually curated but is instead populated online through the policy’s own interactions. During training, the policy model distills successful and failed trajectories into concise natural-language principles paired with its source interaction state , which are then inserted into . To ensure the efficiency of the pipeline, we retain only 50% of the trajectories generated during the GRPO process for distillation. Specifically, we prioritize a balanced retention of successful and failed trajectories; when such a balance cannot be achieved, the remaining quota is filled by randomly sampling from the leftover trajectories.

Deduplication and Pruning. To prevent the memory bank from growing unboundedly and accumulating redundant entries (e.g., near-identical principles arising from GRPO group sampling), we apply curation at three stages. (1) Write-time deduplication: before inserting a new memory entry, we perform an embedding-based nearest-neighbor probe; if its maximum cosine similarity to any existing entry exceeds the threshold (default ), the insertion is skipped. In practice, we enable it for WebShop and disable it for ALFWorld. (2) Retrieval-time deduplication: at inference, we first retrieve a larger candidate pool from bank, then apply greedy embedding-based deduplication before truncating to the top- results, ensuring the returned memories are not semantically redundant. (3) Capacity and quality maintenance: we adopt an EvolveR-style Laplace-smoothed utility score , and periodically prune low-utility entries whose score falls below the pruning threshold (default ) once they have been retrieved at least min_uses_before_prune times (default 3).

A.4 Cold-Start Dataset Construction

We construct a small cold-start dataset to familiarize the policy with the interaction protocol, particularly how to initiate memory retrieval and summarize trajectories into reusable memories. For each benchmark, we collect seed interaction trajectories from the AgentGym dataset Xi et al. (2024).

The cold-start dataset contains two subsets, each comprising 200 examples per benchmark. First, we construct memory-augmented interaction trajectories. For each seed trajectory, we randomly sample an intermediate decision points and insert a memory-retrieval turn. GPT-5.1 generates a context-appropriate retrieval query from the task description and interaction prefix at the selected point. The retrieved memory is produced by asking GPT-5.1 to distill the corresponding source trajectory into concise, reusable guidance. The resulting trajectory therefore demonstrates the complete interaction protocol, including issuing a retrieval request, receiving an experience from the memory bank, and continuing environment interaction in the required output format.

Second, we construct 200 trajectory-to-memory summarization pairs. Each pair takes a completed AgentGym trajectory as input and uses its GPT-5.1 generated summary as the target, following the structured memory schema described in Appendix C. These examples teach the policy to convert interaction histories into compact memories containing a situational precondition and a reusable principle.

We combine the two subsets and perform supervised fine-tuning for 2 epochs with a learning rate of and an effective batch size of 2. GPT-5.1 is used only for offline cold-start data construction and is not involved in RL training or evaluation. The purpose of this stage is protocol and format alignment rather than task-skill acquisition; consistent with this role, the cold-start model alone achieves limited task performance, while the actual reconstruction and decision-making capabilities are subsequently learned through task-level reinforcement learning.

Appendix B Counterfactual Probe Construction

To evaluate whether MemHarness genuinely relies on fine-grained state comparison rather than superficial pattern matching, we constructed a counterfactual probe dataset comprising 2,000 examples (1,000 for ALFWorld and 1,000 for WebShop). For each instance, we sample a real interaction state () and a genuinely retrieved memory () that is highly applicable. We then use a strong LLM to minimally edit the state to produce a counterfactual state (), rendering either inapplicable or misleading.

B.1 Generation Prompt

We use GPT-5.1 to generate the counterfactual states. The prompt (Appendix C) enforces that the edits remain minimal and structurally identical to the original observations, preventing the agent from relying on formatting artifacts to reject the memory.

B.2 Examples of State Edits

Below we provide concrete examples of the factual edits for both benchmarks. In these examples, the MemHarness agent successfully accepts the memory under the original state () but adaptively rewrites or rejects it when presented with the minimal edit ().

ALFWorld Example: Premise Invalidation.

In ALFWorld, memories often provide search heuristics based on past failures. The factual edit invalidates the heuristic’s premise by revealing the target object.

• •

  Task: heat some mug and put it in coffeemachine.

• •

  Retrieved Memory (): If you cannot find a mug in nearby cabinets or drawers after opening several of them, shift your search to visible surfaces like countertops or the sinkbasin.

• •

  Original State (): On the drawer 5, you see nothing.

• •

  Edited State (): On the drawer 5, you see a mug.

• •

  Agent Behavior Shift: In , the agent accepts the memory and redirects its search to open surfaces. In , the premise (failing to find a mug) is broken; the agent rejects the memory (<EMPTY>) and directly picks up the mug.

WebShop Example: Constraint Violation.

In WebShop, memories often recommend finalizing a purchase when the product matches all instructions. The edit subtly alters a product attribute, violating the user’s initial constraint.

• •

  Task: Find me machine wash men’s t-shirts … and price lower than 50.00 dollars.

• •

  Retrieved Memory (): After selecting all required attributes … and confirming the price constraint is satisfied, finish the task by clicking the “Buy Now” button.

• •

  Original State (): … “Price: $19.99” [SEP] “Rating: N.A.” … “Buy Now”

• •

  Edited State (): … “Price: $59.99” [SEP] “Rating: N.A.” … “Buy Now”

• •

  Agent Behavior Shift: In , the agent follows the memory to click “Buy Now”. In , noticing the price now exceeds the $50 limit, the agent rejects the memory to prevent a wrong purchase and navigates back to search.

Appendix C Prompt Templates

This section presents the prompt templates used for memory retrieval, state-conditioned memory reconstruction, agent–environment interaction, trajectory summarization, and fallback reasoning when no adapted memory. Text enclosed in braces denotes a runtime placeholder.

Environment feedback Prompt for Agent

The agent system prompts specify the environment context, interaction history, admissible actions, and the required reasoning–action output format. The templates for ALFWorld and WebShop are presented in Figures 5 and 6, respectively. The memory retrieval instruction guides the agent to issue a retrieval query when additional experience is needed, while preventing retrieval and environment actions from appearing in the same response.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Self-Reasoning Fallback Prompt

When no retrieved principle is applicable, the fallback prompt instructs the agent to rely on the current observation and its own reasoning instead. The corresponding prompt is shown in Figure 7.

媒体内容 · 前往原文查看

Contextual Memory Reconstruction Prompt

The reconstruction prompt adapts a retrieved historical principle to the current task and state, returning <EMPTY> when the principle is not applicable. The full template is presented in Figure 8.

媒体内容 · 前往原文查看

Trajectory Summarization Prompts

After an episode is completed, the trajectory summarization prompt extracts concise and reusable memories grounded in the interaction trajectory. Its required JSON output schema and extraction constraints are provided in Figure 9.

媒体内容 · 前往原文查看

Counterfactual Probe Construction Prompts

This section details the prompts used to construct the counterfactual probes for the analysis in Section 4.5. The complete template is shown in Figure 10 and Figure 11.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Appendix D Case Studies

To intuitively illustrate how MemHarness bridges the gap between historical knowledge and current interaction states, we provide two real reconstruction cases from our evaluation. These examples demonstrate why verbatim replay of past experiences often fails and how our policy actively rewrites memory to provide state-grounded guidance.

D.1 ALFWorld: Abstracting and Transferring Skills

In ALFWorld, verbatim replay of the retrieved memory would introduce hallucinated objects (“credit card” and “coffee table”) into the context, likely causing the agent to output an invalid action. As shown in Figure 12, MemHarness instead extracts the underlying procedural skill (placing a held item at the final destination after a prerequisite step) and explicitly grounds it with the current entities (“cup” and “sidetable”).

媒体内容 · 前往原文查看

D.2 WebShop: Grounding Heuristics into Actionable Targets

In WebShop, the retrieved memory is often an abstract heuristic (e.g., advising the agent to cross-check specific constraints before clicking). A naive replay would simply append this abstract rule to the prompt, offering no concrete operational help. As shown in Figure 13, MemHarness executes this heuristic during the reconstruction phase: it scans the current noisy observation, aligns it with the user’s constraints, and directly outputs the exact target ID to act upon.

媒体内容 · 前往原文查看