Zhiyuan Yao

Yuxin Chen

Zhengxi Lu

Zishan Xu

Yueqing Sun

Yifu Guo

Yuquan Lu

Zhengzhou Cai

Kangning Zhang

Zhuowen Han

Zihan Wang

Ziang Ye

Weiwen Liu

Yongliang Shen

Abstract

Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to skill learning either focus on repeated attempts of one task or use pipelines with multiple stages that entangle extraction, retrieval, and execution. We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks. SkillRise organizes related instances into progressively challenging sequences and uses a single policy to alternate between task solving and curating an evolving skill document passed directly to the next task. Decoupled credit assignment across tasks supervises solving with the current task outcome and curation with discounted downstream outcomes. Experiments on ALFWorld, WebShop, and ScienceWorld show that SkillRise achieves the strongest Pass@1 performance among the compared methods, with gains over the strongest baseline ranging from 2.3 to 8.5 percentage points. Although trained across distinct tasks, its learned curation policy remains effective for repeated attempts on the same task. Further analysis reveals scaling at test time across tasks: performance improves with longer sequences of related tasks even when each task is attempted only once. This trend suggests that SkillRise reuses transferable skills across tasks rather than benefiting from repeated sampling of the same task. SkillRise further retains strong performance while substantially reducing the runtime overhead of skill learning pipelines with multiple stages. Together, these results provide a simple and efficient training paradigm for LLM agents to extract, refine, and reuse transferable skills across tasks. Our code is publicly available at https://github.com/Within-yao/SkillRise.

1 Introduction

Large language model agents are increasingly deployed to solve complex, long-horizon tasks (Anthropic, 2025; OpenAI, 2025; Guo and others, 2025; Xie et al., 2024). In practice, they often encounter streams of related yet distinct tasks that share underlying regularities and reusable solution patterns (Zheng et al., 2025; Ouyang et al., 2026). However, standard agentic RL typically treats each task as an independent episode, discarding the experience acquired during interaction and forcing the agent to repeatedly explore from scratch (Qiu et al., 2026). An ideal agent should not only solve the task at hand, but also extract transferable skills from experience and continually reuse and refine them across subsequent tasks, becoming increasingly capable over time.

Recent work has begun to use reinforcement learning to train agents to extract reusable experience or skills from interaction trajectories (Zhang et al., 2026a; Shi et al., 2026; Wang et al., 2026a). One line repeatedly attempts the same task and summarizes previous trajectories to improve subsequent attempts (Jiang et al., 2025); however, the resulting knowledge can remain instance-specific, requires additional interactions, and is not directly optimized for cross-task transfer. Another line maintains an external experience or skill bank, updating it with newly acquired knowledge and retrieving relevant skills to guide future tasks (Xia et al., 2026; Wu et al., 2025). However, because downstream performance is jointly determined by skill extraction, retrieval, and execution, it is difficult to attribute success or failure to the quality of the extracted skills themselves.

These limitations motivate a simple, end-to-end formulation of cross-task skill learning. We consider a sequence of related yet distinct tasks, in which an agent extracts skills from earlier interactions and applies them to subsequent tasks. Within this sequence, the same policy incorporates each interaction trajectory into an evolving skill document and directly uses the document for the next task, avoiding a complex external skill-management pipeline. Since the tasks share common regularities while differing in their specific instances, performance on later tasks provides a natural measure of skill transferability and enables the policy to jointly learn to produce and use transferable skills. To optimize these two behaviors effectively, we further argue that supervision should follow their temporal roles, with task solving evaluated by the current task outcome and skill curation by performance on subsequent tasks.

Building on this formulation, we introduce SkillRise, an end-to-end reinforcement learning framework for cross-task skill learning. SkillRise first constructs progressively challenging task sequences by selecting similar yet distinct instances from the same task family and ordering them by difficulty, allowing experience from earlier tasks to support later ones. During each rollout, a single policy alternates between solving the current task with an evolving skill document and curating the document based on the resulting trajectory before proceeding to the next task instance. SkillRise employs decoupled cross-task credit assignment that assigns the current task reward to task solving and a discounted return over subsequent task rewards to skill curation. Group-relative advantages are then computed over trials sharing the same task group, sequence stage, and behavioral phase. Together, these designs enable the policy to continually extract, refine, and reuse transferable skills while solving a sequence of tasks.

We evaluate SkillRise on ALFWorld, WebShop, and ScienceWorld. Across all three agentic benchmarks, SkillRise consistently outperforms prompting-based methods and standard reinforcement learning baselines. Its advantage over methods that repeatedly attempt the same task indicates that the extracted skills transfer across different task instances. More importantly, SkillRise exhibits cross-task test-time scaling, improving as it encounters longer sequences of related tasks at test time and progressively refines its skill document. Competing methods do not show this growth trend. These results show that SkillRise learns a transferable mechanism for cross-task self-improvement, allowing skills acquired from earlier tasks to systematically improve performance on later ones.

Overall, our contributions can be summarized as follows:

• •

  We formulate cross-task skill learning over ordered sequences of related yet distinct task instances, where later-task outcomes directly evaluate whether skills curated from earlier interactions transfer to subsequent tasks.

• •

  We develop SkillRise, an end-to-end reinforcement learning framework that uses a single policy to solve tasks and curate an evolving skill document, together with decoupled cross-task credit assignment and role-aware group-relative optimization.

• •

  Extensive experiments on ALFWorld, WebShop, and ScienceWorld show that SkillRise achieves the best overall results among the compared methods. Further analyses demonstrate within-task generalization and cross-task test-time scaling, and show that its compact end-to-end design offers a better balance of effectiveness and efficiency than multi-stage skill-learning pipelines.

2 Preliminaries

We consider an LLM agent that interacts with an environment to solve a task instruction sampled from a task distribution . At each step , the agent receives an observation and selects an action according to

(1)

where denotes the agent policy and is the interaction history available to the agent. After executing , the environment returns the next observation .

An episode produces a trajectory and terminates when the task is completed or the interaction budget is exhausted. The trajectory is evaluated by a verifiable outcome reward . The standard reinforcement learning objective is to maximize the expected task reward:

(2)

This objective treats each task independently; in the following section, we extend it to a sequence of related tasks connected through an evolving skill state.

3 Method

Figure 1 presents an overview of SkillRise. The framework first organizes related task instances into a progressive sequence, then uses a single policy to alternate between task solving and skill curation along the sequence. The two roles are optimized with temporally aligned learning signals to jointly improve task-solving and cross-task skill-learning capabilities.

3.1 Cross-Task Sequence Construction

Standard agentic reinforcement learning treats each task as an independent episode, preventing skills learned from one task from being transferred to another. To create a learning signal for cross-task skill transfer, we organize tasks into ordered sequences

(3)

where is a distribution over sequences of distinct instances from the same task family. Although these instances differ in their concrete entities and goals, they share common interaction routines that can be abstracted into transferable skills. Consequently, skills learned from earlier tasks can guide later tasks, while performance on the later tasks provides a direct measure of whether those skills transfer beyond a single instance.

To construct the sequences, we first partition tasks by environment-provided family metadata, and then greedily group diverse instances within each family. We arrange the selected tasks from relatively simple to more complex according to their task attributes. For example, we group WebShop tasks by product category and order distinct requests by the number of required attributes and options. The resulting tasks are executed sequentially and connected through an evolving skill document.

3.2 Cross-Task Rollout with Skill Evolution

Given a cross-task sequence , SkillRise performs a sequential rollout while maintaining a single textual skill document that summarizes the transferable skills acquired through the first tasks. Each trial starts from an empty document . For the -th task, the policy first uses the current document to generate a task-solving trajectory and receive its task reward:

(4)

For every task except the last, the same policy then switches from task solving to skill curation and produces a revised document:

(5)

The two phases share the same policy parameters and differ only in their role-specific instructions.

During curation, the policy rewrites the complete document: it preserves useful skills from , consolidates successful procedures or failure modes revealed by , and removes instance-specific details. Only the revised document is provided to task ; earlier task trajectories are not carried into its solving context. Hence, is the sole information channel across tasks, and its utility is reflected by performance on subsequent tasks. During training, we sample independent trials for each sequence, with every trial starting from an empty document and evolving its own skill document.

3.3 Decoupled Credit Assignment and Optimization

Decoupled credit assignment.

Task solving and skill curation play different temporal roles. A solving trajectory can be evaluated directly by the current task reward, whereas the revised skill document is produced only after the task terminates and can influence only subsequent tasks. Assigning the same sequence-level return to both phases would therefore conflate task-solving quality with skill transferability. Let index the independent trials and denote the phase. We assign the phase-level return

(6)

where is the cross-task discount factor. Thus, solving is supervised only by the current task outcome, while curation is evaluated by the discounted outcomes of later tasks. The discount places more credit on nearby tasks, whose performance is more directly affected by the revised document.

Role-aware group-relative optimization.

For a fixed sequence, task position , and phase , we compare the returns of its trials:

(7)

Grouping by both task position and phase ensures that solving and curation returns never serve as baselines for one another, while also accounting for differences across positions in the sequence.

Let denote the token-level importance ratio between the current and old policies for token . We jointly optimize all solving and curation responses using the clipped objective

(8)

where is the clipping range. The phase-level advantage is applied to all policy-generated tokens in the corresponding phase. Since the two phases share the same policy, this objective jointly improves task solving and skill curation while keeping their learning signals separated.

4 Experiments

4.1 Experimental Setup

Benchmarks.

We evaluate on three interactive text environments. ALFWorld (Shridhar et al., 2020) requires an embodied household agent to navigate and manipulate objects; we use its TextWorld version and report results for all six task families: Pick, Look, Clean, Heat, Cool, and Pick2. WebShop (Yao et al., 2022a) requires an agent to search for, configure, and purchase a product that satisfies a natural-language request. We use the 1,000-product version with synthetic goals. ScienceWorld (Wang et al., 2022) evaluates grounded scientific reasoning through interactive experiments. We evaluate on 128 held-out task instances from each environment.

Baselines.

We compare against three prompting baselines: Zero-shot, ReAct (Yao et al., 2022b), and Reflexion (Shinn et al., 2023). We further include four task-independent RL algorithms: PPO (Schulman et al., 2017), RLOO (Ahmadian et al., 2024), GRPO (Shao et al., 2024), and GiGPO (Feng et al., 2025). Finally, we compare with LaMer (Jiang et al., 2025), a Meta-RL baseline that adapts across repeated attempts of the same task through textual reflection.

Implementation Details.

All methods use Qwen3-1.7/4B (Yang et al., 2025) as the backbone. For SkillRise, each training batch contains 16 sequences, each with tasks and independent trials. LaMer also uses eight trials with three attempts, whereas the task-independent RL baselines use 24 rollouts per task. Thus, every trainable method processes the same 384 task plays per update. We use an actor learning rate of , a mini-batch size of 128, a maximum response length of 1,024 tokens, and train for up to 150 updates. For SkillRise, the cross-task discount factor is set to . Evaluation uses sampling with temperature 0.7. Training is implemented with FSDP and vLLM, and all experiments are conducted on eight NVIDIA H800 GPUs.

媒体内容 · 前往原文查看

	ALFWorld							WebShop	ScienceWorld
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	SR	SR
Zero-shot	1.5	6.1	3.6	4.2	2.8	8.6	4.4	1.4	0.8
ReAct	45.5	30.3	21.4	12.5	18.1	23.5	25.0	2.1	7.8
Reflexion	74.2	39.4	19.0	25.0	13.9	13.6	28.9	2.7	7.8
PPO	82.1	70.0	57.9	52.9	50.0	72.0	67.2	74.2	39.1
RLOO	88.2	37.5	96.1	61.9	80.0	42.1	74.2	76.6	43.0
GRPO	94.1	75.0	96.2	66.7	75.0	52.6	79.7	75.8	42.2
GiGPO	90.3	88.2	96.4	83.3	75.0	60.0	83.6	77.3	46.1
LaMer	88.2	62.5	96.2	71.4	75.0	42.1	76.7	74.2	41.4
SkillRise	100.0	87.5	88.5	81.0	80.8	68.4	85.9	84.4	54.6

媒体内容 · 前往原文查看

	ALFWorld			WebShop			ScienceWorld
Method	Pass@1	Pass@2	Pass@3	Pass@1	Pass@2	Pass@3	Pass@1	Pass@2	Pass@3
Zero-shot	4.4	9.4	12.5	1.4	2.1	2.3	0.8	1.6	2.3
ReAct	25.0	45.3	57.0	2.1	4.5	4.5	7.8	10.2	12.5
Reflexion	28.9	36.7	42.2	2.7	3.3	3.5	7.8	12.5	17.2
PPO	67.2	69.5	73.4	74.2	75.8	75.8	39.1	45.3	47.7
RLOO	74.2	80.5	82.8	76.6	78.1	78.9	43.0	48.4	52.3
GRPO	79.7	81.3	82.8	75.8	76.6	78.1	42.2	50.0	50.0
GiGPO	83.6	86.7	89.1	77.3	79.7	80.5	46.1	50.8	55.5
LaMer	76.7	86.7	91.4	74.2	87.5	94.5	41.4	46.1	46.8
SkillRise	85.9	91.4	92.2	84.4	93.8	96.1	54.6	58.5	61.0

4.2 Main Results

Overall Performance.

As shown in Table 1, SkillRise achieves the strongest overall Pass@1 performance across all three benchmarks. It obtains 85.9% on ALFWorld, 84.4% on WebShop, and 54.6% on ScienceWorld, outperforming the strongest baseline, GiGPO, by 2.3, 7.1, and 8.5 percentage points, respectively. On ALFWorld, SkillRise ranks first or second in five of the six task families, leading to the best overall success rate despite substantial variation across task types. We attribute these gains to cross-task training, where related but distinct instances repeatedly expose the policy to different realizations of the same underlying interaction routines. This structured training signal makes transferable regularities easier to identify and reinforce across tasks, accelerating their internalization into the model parameters and enabling the policy to acquire reusable task-solving skills more efficiently.

Generalization to Within-Task Adaptation.

Although SkillRise learns to curate skills from sequences of distinct tasks during training, we evaluate Pass@2 and Pass@3 by retrying the same held-out task while carrying forward the evolving skill document. This protocol tests whether the skill-learning strategy acquired across tasks can generalize to within-task adaptation. As shown in Table 2, SkillRise achieves the best Pass@2 and Pass@3 on all three benchmarks. At Pass@3, it reaches 92.2%, 96.1%, and 61.0% on ALFWorld, WebShop, and ScienceWorld, exceeding the strongest baseline by 0.8, 1.6, and 5.5 percentage points, respectively. Notably, it also outperforms LaMer, which is trained explicitly on repeated attempts of the same task, by 0.8, 1.6, and 14.2 points. These results suggest that the learned curation policy captures a general strategy for converting interaction outcomes into actionable skills, which transfers beyond its cross-task training setting to guide subsequent attempts on the same task.

5 Analysis

5.1 Cross-Task Test-Time Scaling

We investigate whether the cross-task skill-learning capability acquired during training scales with the amount of related task context available at test time. We partition the same held-out task set into sequences of tasks and allow SkillRise to carry forward and update its skill document within each sequence. As shown in Figure 2, SkillRise improves monotonically over the evaluated sequence lengths, from 83.6% at to 87.5% at . In contrast, LaMer, GiGPO, and GRPO show no sustained improvement as grows, and the margin over the strongest baseline widens from 3.9 to 8.6 percentage points. Since every task receives only one attempt, this gain does not result from additional sampling on the same instance. Instead, the trend suggests that the trained policy can transform interactions with earlier tasks into transferable skills for later tasks, with the benefit becoming more pronounced as the skill document is progressively refined over longer sequences.

5.2 Performance across Model Sizes

媒体内容 · 前往原文查看

	Qwen3
Method	1.7B	4B
GRPO	75.0	79.7
LaMer	73.4	76.7
SkillRise	78.1	85.9

We further evaluate whether the gains of SkillRise extend beyond a single model size. As shown in Table 3, SkillRise achieves the strongest Pass@1 performance at both evaluated scales. With Qwen3-1.7B, SkillRise reaches 78.1%, exceeding the strongest baseline by 3.1 percentage points. With Qwen3-4B, it further improves to 85.9%, widening this margin to 6.2 points. Thus, the advantage of cross-task skill learning is already present with the smaller backbone and becomes more pronounced at the larger scale.

All three methods benefit from increasing the backbone size, but the magnitude of improvement differs substantially. SkillRise gains 7.8 points from 1.7B to 4B, compared with 4.7 points for GRPO and 3.3 points for LaMer. Unlike task-independent optimization, SkillRise must jointly learn to solve the current task, identify regularities shared across related tasks, and curate them into a reusable skill document. The larger gain suggests that additional model capacity is particularly useful for this combination of task execution and cross-task abstraction. Overall, these results show that the effectiveness of SkillRise is not confined to a single parameter scale and that, within the evaluated range, its advantage increases as the backbone grows.

5.3 Ablation Studies

Sensitivity to Cross-Task Discounting.

We vary the cross-task discount factor over while keeping the remaining settings fixed. As shown in the left panel of Figure 3, all four variants follow closely aligned learning trajectories, exhibiting comparable improvements throughout training and converging within approximately one percentage point of one another. This consistency shows that the gains of SkillRise do not rely on carefully tuning the relative weight assigned to future tasks. Instead, the cross-task learning signal remains effective over a broad range of discount factors, indicating that the proposed credit assignment is robust to reasonable variations in how strongly downstream outcomes supervise skill curation.

Effect of Skill Curation.

The no-curation variant retains the same task sequences and environment interaction budget, but removes the inter-task curation phase and prevents skill transfer between tasks. As shown in the right panel of Figure 3, the three methods improve similarly during the early stage, but SkillRise begins to separate after sufficient cross-task interactions have accumulated and maintains the strongest training score thereafter. At the end of training, it leads the no-curation variant by roughly three percentage points and GRPO by more than six points. The advantage over the no-curation variant shows that training on related task sequences alone is insufficient. Explicitly transforming completed trajectories into an evolving skill document allows useful procedures discovered on earlier tasks to be consolidated and reused on later ones. The additional improvement over task-independent GRPO further suggests that cross-task skill accumulation provides a complementary learning signal beyond optimizing each task in isolation.

Overall, these ablations confirm the importance of skill curation while showing that SkillRise remains robust to the choice of cross-task discount factor. Together, they support the effectiveness of the complete framework.

5.4 Pipeline Complexity and Training Efficiency

Existing skill-learning methods often decompose skill acquisition into a multi-stage management pipeline. RetroAgent generates reflections, stores them in a growing memory buffer, retrieves relevant lessons, and updates their utilities (Zhang et al., 2026b). SkillRL further relies on teacher-based trajectory distillation, hierarchical SkillBank construction, cold-start SFT, skill retrieval, and iterative bank refinement (Xia et al., 2026). For our SkillRL reproduction, we use Gemini-2.5-Pro as the teacher model. In such pipelines, downstream performance is jointly determined by skill extraction, storage, retrieval, updating, and execution. This coupling makes it difficult to attribute success or failure to the learned skill itself, while the additional stages also introduce substantial computational overhead.

SkillRise instead adopts a compact end-to-end training paradigm: the same policy solves each task, curates a single sequence-local skill document, and directly uses it on the next task. This removes the need for an external teacher, separate memory maintenance, and an additional retrieval module. As shown in Figure 4, SkillRise achieves an 85.9% average success rate on ALFWorld, matching RetroAgent and outperforming SkillRL by 12.5 percentage points. Under the same GPU configuration, RetroAgent and SkillRL require and the end-to-end running time of SkillRise, respectively. Thus, transferable skill learning does not require a long external management pipeline. Directly supervising skill curation through subsequent task outcomes provides a clearer learning signal while retaining strong performance with substantially lower training overhead.

6 Related Work

Reinforcement Learning for LLM Agents.

Reinforcement learning has become a central approach for improving LLMs, evolving from preference alignment with human feedback to reinforcement learning with verifiable rewards, which provides scalable supervision for reasoning and improves training efficiency (Shao et al., 2024; Guo and others, 2025; Yao et al., 2026; Chen et al., 2026). As LLMs are increasingly deployed as interactive agents, agentic RL extends this paradigm from single-turn generation to multi-turn decision making, where long horizons and delayed outcomes make exploration and credit assignment substantially harder (Zhang et al., 2025). Existing methods develop hierarchical, step-level, or milestone-guided credit assignment (Feng et al., 2025; Zhang et al., 2026c; He et al., 2026; Wang et al., 2026b; Lu et al., 2026a), improve exploration through autonomous environment interaction and online curricula, including for web navigation and long-context software engineering (Ye et al., 2026; Qi et al., 2025; Wei et al., 2025; Wang et al., 2025), or provide scalable training abstractions for heterogeneous agent workflows (Luo et al., 2025; Xi et al., 2025). These advances demonstrate that outcome-based supervision can improve long-horizon interaction, but most methods still optimize independently sampled task episodes. Consequently, how an agent can learn from the relationship among episodes and turn earlier interactions into reusable procedural knowledge remains less explored.

Experience and Skill Learning in LLM Agents.

Learning from past interactions is a central mechanism for self-evolving agents, with the core challenge being how to transform interaction histories into reusable knowledge (Gao et al., 2025). Concrete representations retain trajectories, examples, or verbal reflections for later reuse (Shinn et al., 2023; Zhao et al., 2024). More abstract representations distill histories into guidelines, workflows, or reasoning strategies that can generalize across instances (Fu et al., 2024; Wang et al., 2024; Ouyang et al., 2025). Recently, editable skill artifacts have emerged as a modular form of procedural memory, allowing agents to consolidate, revise, and reuse high-level solution procedures (Ni et al., 2026; Yang et al., 2026). Reinforcement learning is increasingly used to train different stages of this skill lifecycle, including skill utilization, internalization, extraction, and curation (Xia et al., 2026; Lu et al., 2026b; Ouyang et al., 2026). LaMer formulates adaptation as a cross-episode Meta-RL problem: the agent repeatedly interacts with the same task and conditions later attempts on environment feedback and textual reflections (Jiang et al., 2025). For skill evolution, Evolving-RL jointly optimizes skill extraction and skill-conditioned solving by evaluating extracted skills on related downstream tasks (Fan et al., 2026). SkillOS keeps the executor frozen and trains a separate curator to update a skill repository, using later performance over related task streams as delayed supervision (Ouyang et al., 2026). These methods manage skill extraction, storage, retrieval, and execution in different ways. However, multi-stage pipelines can make it difficult to attribute downstream performance to individual components.

7 Conclusion

We introduced SkillRise, an end-to-end reinforcement learning framework for cross-task skill learning. It organizes related yet distinct tasks into ordered sequences, where a single policy alternates between task solving and curating an evolving skill document. Decoupled credit assignment uses current-task outcomes for solving and downstream outcomes for skill curation, aligning supervision with their temporal roles. Across ALFWorld, WebShop, and ScienceWorld, SkillRise achieves the strongest overall performance among the compared methods, generalizes from cross-task training to repeated attempts, and improves with longer related-task sequences at test time. Moreover, the pipeline comparison shows that SkillRise maintains strong performance while requiring only one sixth of the running time. Overall, our work provides a simple and efficient training paradigm for enabling LLM agents to extract, refine, and reuse transferable skills across tasks.

8 Limitation

Our current formulation assumes access to task-family metadata for constructing sequences of related instances; automatically discovering such relationships in open-ended task streams remains an important direction. Moreover, due to computational constraints, our experiments are limited to models of up to 4B parameters, leaving evaluation at larger model scales for future work. Finally, our experiments focus on three text-based agent benchmarks and verifiable outcome rewards. Evaluating the framework on broader multimodal, real-world, and less readily verifiable tasks would further establish its generality.

References

• A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker (2024) Back to basics: revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267. Cited by: §4.1.
• Anthropic (2025) Claude sonnet 4.5 model card. External Links: Link Cited by: §1.
• Y. Chen, Y. Wang, Y. Zhang, Z. Ye, Z. Cai, Y. Shi, Q. Gu, H. Su, X. Cai, X. Wang, et al. (2026) Learning to self-verify makes language models better reasoners. arXiv preprint arXiv:2602.07594. Cited by: §6.
• Z. Fan, W. Jin, F. Zhang, B. Li, Y. Dong, Y. Hu, and J. Li (2026) Evolving-rl: end-to-end optimization of experience-driven self-evolving capability within agents. arXiv preprint arXiv:2605.10663. Cited by: §6.
• L. Feng, Z. Xue, T. Liu, and B. An (2025) Group-in-group policy optimization for llm agent training. External Links: 2505.10978, Link Cited by: §4.1, §6.
• Y. Fu, D. Kim, J. Kim, S. Sohn, L. Logeswaran, K. Bae, and H. Lee (2024) Autoguide: automated generation and selection of context-aware guidelines for large language model agents. Advances in Neural Information Processing Systems 37, pp. 119919–119948. Cited by: §6.
• H. Gao, J. Geng, W. Hua, M. Hu, X. Juan, H. Liu, S. Liu, J. Qiu, X. Qi, Y. Wu, et al. (2025) A survey of self-evolving agents: what, when, how, and where to evolve on the path to artificial super intelligence. arXiv preprint arXiv:2507.21046. Cited by: §6.
• D. Guo et al. (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §6.
• S. He, L. Feng, Q. Wei, X. Cheng, L. Feng, and B. An (2026) Hierarchy-of-groups policy optimization for long-horizon agentic tasks. arXiv preprint arXiv:2602.22817. Cited by: §6.
• Y. Jiang, L. Jiang, D. Teney, M. Moor, and M. Brbic (2025) Meta-rl induces exploration in language agents. arXiv preprint arXiv:2512.16848. Cited by: §1, §4.1, §6.
• Z. Lu, Z. Yao, Z. Han, Z. Wang, J. Wu, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, et al. (2026a) Self-distilled agentic reinforcement learning. arXiv preprint arXiv:2605.15155. Cited by: §6.
• Z. Lu, Z. Yao, J. Wu, C. Han, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen (2026b) Skill0: in-context agentic reinforcement learning for skill internalization. arXiv preprint arXiv:2604.02268. Cited by: §6.
• X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang (2025) Agent lightning: train any ai agents with reinforcement learning. arXiv preprint arXiv:2508.03680. Cited by: §6.
• J. Ni, Y. Liu, X. Liu, Y. Sun, M. Zhou, P. Cheng, D. Wang, E. Zhao, X. Jiang, and G. Jiang (2026) Trace2skill: distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158. Cited by: §6.
• OpenAI (2025) Introducing gpt-5. External Links: Link Cited by: §1.
• S. Ouyang, J. Yan, Y. Chen, R. Han, Z. Wang, B. D. Mishra, R. Meng, C. Li, Y. Jiao, K. Zha, et al. (2026) Skillos: learning skill curation for self-evolving agents. arXiv preprint arXiv:2605.06614. Cited by: §1, §6.
• S. Ouyang, J. Yan, I. Hsu, Y. Chen, K. Jiang, Z. Wang, R. Han, L. T. Le, S. Daruki, X. Tang, et al. (2025) Reasoningbank: scaling agent self-evolving with reasoning memory. arXiv preprint arXiv:2509.25140. Cited by: §6.
• Z. Qi, X. Liu, I. L. Iong, H. Lai, X. Sun, J. Sun, X. Yang, Y. Yang, S. Yao, W. Xu, et al. (2025) Webrl: training llm web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, Vol. 2025, pp. 79791–79821. Cited by: §6.
• L. Qiu, Z. Gao, J. Chen, Y. Ye, W. Huang, X. Xue, W. Qiu, and S. Tang (2026) AutoRefine: from trajectories to reusable expertise for continual llm agent refinement. arXiv preprint arXiv:2601.22758. Cited by: §1.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §4.1.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §4.1, §6.
• Y. Shi, Y. Chen, Z. Lu, Y. Miao, S. Liu, Q. Gu, X. Cai, X. Wang, and A. Zhang (2026) Skill1: unified evolution of skill-augmented agents via reinforcement learning. arXiv preprint arXiv:2605.06130. Cited by: §1.
• N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, pp. 8634–8652. Cited by: §4.1, §6.
• M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2020) Alfworld: aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768. Cited by: §4.1.
• J. Wang, Q. Yan, Y. Wang, Y. Tian, S. S. Mishra, Z. Xu, M. Gandhi, P. Xu, and L. L. Cheong (2026a) Reinforcement learning for self-improving agent with skill library. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1529–1550. Cited by: §1.
• R. Wang, P. Jansen, M. Côté, and P. Ammanabrolu (2022) Scienceworld: is your agent smarter than a 5th grader?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11279–11298. Cited by: §4.1.
• Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, et al. (2025) Ragen: understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. Cited by: §6.
• Z. Wang, Y. Yan, H. Li, T. Pan, D. Li, R. Zhang, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen (2026b) Milestone-guided policy learning for long-horizon language agents. arXiv preprint arXiv:2605.06078. Cited by: §6.
• Z. Z. Wang, J. Mao, D. Fried, and G. Neubig (2024) Agent workflow memory. arXiv preprint arXiv:2409.07429. Cited by: §6.
• Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, and S. I. Wang (2025) SWE-rl: advancing llm reasoning via reinforcement learning on open software evolution. External Links: 2502.18449, Link Cited by: §6.
• R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang, et al. (2025) Evolver: self-evolving llm agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079. Cited by: §1.
• Z. Xi, J. Huang, C. Liao, B. Huang, H. Guo, J. Liu, R. Zheng, J. Ye, J. Zhang, W. Chen, et al. (2025) Agentgym-rl: training llm agents for long-horizon decision making through multi-turn reinforcement learning. arXiv preprint arXiv:2509.08755. Cited by: §6.
• P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. (2026) Skillrl: evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234. Cited by: §1, §5.4, §6.
• T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. (2024) Osworld: benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37, pp. 52040–52094. Cited by: §1.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1.
• Y. Yang, Z. Gong, W. Huang, Q. Yang, Z. Zhou, Z. Huang, Y. Li, X. Gao, Q. Dai, B. Liu, et al. (2026) Skillopt: executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904. Cited by: §6.
• S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022a) Webshop: towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems 35, pp. 20744–20757. Cited by: §4.1.
• S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2022b) React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §4.1.
• Z. Yao, Y. Zhang, Y. Chen, Y. Sun, Z. Xu, Y. Yang, T. Hu, Q. Gu, H. Su, and X. Cai (2026) CoBA-rl: capability-oriented budget allocation for reinforcement learning in llms. arXiv preprint arXiv:2602.03048. Cited by: §6.
• Z. Ye, W. Shi, Y. Liu, Y. Wang, Z. Cai, Y. Shi, Q. Gu, X. Cai, and F. Feng (2026) Look before you leap: autonomous exploration for llm agents. arXiv preprint arXiv:2605.16143. Cited by: §6.
• G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, et al. (2025) The landscape of agentic reinforcement learning for llms: a survey. arXiv preprint arXiv:2509.02547. Cited by: §6.
• H. Zhang, Q. Long, J. Bao, T. Feng, W. Zhang, H. Yue, and W. Wang (2026a) MemSkill: learning and evolving memory skills for self-evolving agents. arXiv preprint arXiv:2602.02474. Cited by: §1.
• X. Zhang, Z. Liu, Y. Zhang, X. Hu, and W. Shao (2026b) Retroagent: from solving to evolving via retrospective dual intrinsic feedback. arXiv preprint arXiv:2603.08561. Cited by: §5.4.
• Z. Zhang, K. Song, X. Wang, Y. Hu, W. Yan, et al. (2026c) CM2: reinforcement learning with checklist rewards for multi-turn and multi-step agentic tool use. External Links: 2602.12268, Link Cited by: §6.
• A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang (2024) Expel: llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, pp. 19632–19642. Cited by: §6.
• J. Zheng, X. Cai, Q. Li, D. Zhang, Z. Li, Y. Zhang, L. Song, and Q. Ma (2025) Lifelongagentbench: evaluating llm agents as lifelong learners. arXiv preprint arXiv:2505.11942. Cited by: §1.

Appendix A Prompts

We reproduce the user-message templates used for task solving and skill curation. Braced fields are filled at runtime. At the start of each task sequence, an empty skill document is represented by (The skill document is currently empty. No skills have been distilled yet.)

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Appendix B Case Study

We show selected actions and abridged curation outputs from the first two tasks of two ALFWorld training rollouts with sequence length . Displayed tasks are numbered from 1. Each rollout begins with an empty skill document; between tasks, only the updated document is carried forward, not earlier-task trajectories. Ellipses mark omitted actions; arrows and comments after # are author annotations. Case 1 (Figure 11) pairs a failure-derived rule with the next task conditioned on that rule. Case 2 (Figure 12) shows an incremental document revision. These examples illustrate the update process rather than isolate a causal effect.

Case 1: distilling a lesson from failure.

The first task asks for two target instances, but the agent places only one and fails. The subsequent curation output adds a general instruction to repeat the procedure for each instance. The next task receives this document, places both instances of a different object, and succeeds.

• •

  Task 1: put two spraybottle in cabinet  ( failed)

• •

  Task 2: put two peppershaker in cabinet  ( succeeded)

媒体内容 · 前往原文查看

 the observed failure is summarized in skill v1

 skill v1 is included in the next solve prompt

Case 2: incremental skill evolution.

Both displayed tasks succeed. Task 1 places an object on a destination that requires no open action, and skill v1 contains a basic placement workflow. Task 2 requires opening a closed microwave; afterward, the curation output adds an “open the destination if closed” condition to skill v2. This case documents the revision itself, not downstream use of the new condition.

媒体内容 · 前往原文查看

 a basic placement workflow is recorded in skill v1

 skill v1 is included; Task 2 encounters a closed microwave

 the open-if-closed condition is added to skill v2