# Skill-：用强化学习渐进式生成智能体技能
- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-08-03 08:00
- AIHOT 分数：53
- AIHOT 链接：https://aihot.virxact.com/items/cmse8uace0hssro2e3kcmifng
- 原文链接：https://arxiv.org/abs/2608.01678
## AI 摘要
Skill- 提出一种基于强化学习的智能体技能生成方法，将技能构建分解为可独立评估的序列编辑，并引入回滚奖励，通过对比编辑前后技能在锚定查询上的下游执行效果来分配信用。在 GPT-4o 作为工作模型时，Skill- 在 CL-Bench 和 tau2-bench 上分别将平均下游成功率较最强基线提升 3.3 和 6.7 个百分点。消融实验验证了回滚奖励与渐进式生成的有效性，代码已开源。
## 正文
Junhao Shen
Zhanqiu Zhang
Yiwen Guo
Hong Cheng
shen.junhao@outlook.com
{zqzhang27,guoyiwen89}@gmail.com
hcheng@se.cuhk.edu.hk
Abstract
Recent large language model agents often use external skills as modular procedural units that condition inference and improve complex task solving. Thus, automatically generating high-quality skills from documents or experience has become an important problem. Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill- generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill- improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation. Code is available at https://github.com/ejhshen/skill-alpha.
1 Introduction
Recent large language model (LLM) agents (Luo et al., 2025; Wang et al., 2024) are usually expected to solve tasks that require multi-step reasoning (Plaat et al., 2026), long-horizon planning (Huang et al., 2024), and reliable tool use (Shen, 2024). A practical way to extend these agents is to provide external skills that condition inference and guide how the agent decomposes tasks, invokes tools, and checks intermediate results (Wang et al., 2025a; 2023; Shen et al., 2026). As skills become reusable modules for shaping agent behavior without retraining the model, automatically generating high-quality skills from documents or experience becomes an important problem (Anthropic, 2025).
Existing skill generation methods largely rely on heuristics, prompting, or pipeline-style consolidation. Document-to-skill methods compress documents, rules, or task descriptions into procedural instructions (Si et al., 2026; Yang et al., 2026), while experience-to-skill methods distill successful or failed executions into reusable guidance (Ni et al., 2026; Mi et al., 2026). Although effective in specific settings, these methods usually require separate designs for different evidence sources and provide limited guidance on how each piece of evidence should change the skill being written. Thus, the central challenge is not only to produce skill text, but to learn a unified skill generation process.
However, learning to generate skills is difficult because skills do not have a natural supervision signal based on relevance or correctness. Unlike mathematical reasoning, web search, or memory management, where outputs can often be evaluated against explicit factual answers or execution results (DeepSeek-AI et al., 2025; Jin et al., 2025; Yan et al., 2025), the quality of a skill lacks a similarly direct supervision signal and is often assessed through its effect on downstream task performance. A fluent skill may still be redundant, over-specific, or misleading, while a compact edit may substantially improve the worker agent’s behavior (Huang et al., 2026; Zhou et al., 2026). This makes execution-grounded credit assignment the key obstacle to learning-based skill generation.
To address this issue, we propose Skill-, a reinforcement learning method for progressive agent skill generation. We decompose skill construction into a sequence of local edit decisions, allowing downstream execution feedback to be assigned to individual edits. Specifically, given a skill draft and new evidence, the generator learns whether to add new procedures, revise imprecise rules, merge redundant guidance, remove harmful or over-specific content, or leave the skill unchanged. We introduce rollback reward as the key training signal. After a candidate edit is applied, the original and edited skills are evaluated on the same evidence-related anchored query and compared by a benchmark-specific verifier, so that the reward reflects the local effect of changing the skill condition. In this way, Skill- turns skill generation pipelines into a unified learning problem over local skill edits, enabling the skill generator to be trained with reinforcement learning.
We evaluate Skill- in both document- and experience-to-skill settings. Under the GPT-4o worker, Skill- improves average downstream success rates over the strongest baselines by 3.3 points on CL-Bench (Dou et al., 2026) and 6.7 points on tau2-bench (Barres et al., 2026). Ablations validate the importance of rollback reward and progressive generation, and analysis further shows that progressive generation is robust to evidence reordering while depending more strongly on evidence granularity.
Overall, our contributions are threefold. First, we formulate agent skill generation as a progressive decision-making problem, unifying document-to-skill and experience-to-skill generation under a single learning framework. Second, we introduce rollback reward, which provides execution-grounded credit assignment for local skill edits. Third, we empirically show that Skill- produces effective skills from heterogeneous evidence sources, with ablations and analyses validating the roles of rollback reward and progressive generation.
2 Related Work
Agent Skills. Agent skills are reusable procedural modules that condition the inference process of an agent and reshape its behavior on downstream tasks (Anthropic, 2025). Recent work has studied skills from multiple perspectives, including skill evolution (Shen et al., 2026; Xia et al., 2026; Ouyang et al., 2026), selection (Zeng et al., 2026; Liu et al., 2026; Zheng et al., 2026), utilization (Wang et al., 2023; 2025a), and internalization (Wang et al., 2026b). These studies show that skills are functional units that can guide planning, tool use, and long-horizon task solving (Wang et al., 2026c; d; Xu & Nie, 2025). Because of this role, more attention has shifted toward skill generation and acquisition (Si et al., 2026; Yang et al., 2026; Ni et al., 2026; Mi et al., 2026) and benchmark evaluation (Huang et al., 2026; Zhou et al., 2026; Li et al., 2026). Skill- follows this direction by treating automatic external-skill generation as the target of learning.
Skill Generation and Acquisition. Skill generation and acquisition focus on producing reusable skill modules from evidence sources such as documents, task contexts, experience, and execution trajectories. Early work explored converting verifier feedback into reusable procedural memory, including ExpeL (Zhao et al., 2024) and Agent Workflow Memory (Wang et al., 2025c). Subsequent work has studied automatic skill construction through prompt-based skill creation (Anthropic, 2026). More recent pipeline-based skill generation systems generally fall into two categories. For document-to-skill settings, Ctx2Skill (Si et al., 2026) and AutoSkill (Yang et al., 2026) aim to compress task documents, rules, or contexts into executable procedural instructions; for experience-to-skill settings, Trace2Skill (Ni et al., 2026), SkillX (Wang et al., 2026a), and SkillPro (Mi et al., 2026) distill trajectories, feedback, or hierarchical experience into reusable skill knowledge. These methods automate skill acquisition but rely on heuristic or pipeline-style consolidation tailored to specific evidence types. In contrast, Skill- formulates skill generation as a progressive decision process, enabling one learned generator to operate over heterogeneous evidence sources.
Reinforcement Learning for External Module Optimization. Reinforcement learning (RL) has become a common approach for optimizing LLM agents to enhance slow-thinking reasoning capabilities (Schulman et al., 2017; DeepSeek-AI et al., 2025; Yu et al., 2025; Shen et al., 2025; Cheng et al., 2025). Beyond optimizing the acting policy itself, recent work also applies RL to external module management. In tool-use and web-search settings, RL is used to improve how agents invoke external tools (Li et al., 2025; Singh et al., 2025), interact with environments (Cheng et al., 2025; Zhao et al., 2026), and use search results (Qi et al., 2024; Jin et al., 2025) to complete complex tasks. Another line of work uses RL to optimize external memory operations, including Memory-R1 (Yan et al., 2025) and Mem- (Wang et al., 2025b). However, skill generation differs from both tool-use optimization and memory management because tool-use and memory-management decisions can often be judged by factual correctness or relevance. By contrast, a skill is a procedural module whose value is expressed largely through how it changes the worker agent’s future behavior. Skill- addresses this problem by introducing rollback reward for skill generation, which compares local skill edits against the current skill on the same anchored query and trains the generator with an execution-grounded signal.
3 Preliminaries
LLM Agent. We model a worker agent as a fixed policy that interacts with an environment to solve a task. Given a task instruction or context , the worker produces an execution trajectory , where and denote the observation and action at step . An external skill is inserted into the worker context. The skill-conditioned worker induces the trajectory distribution . In this work, is not the training target. We instead train a skill-editing policy that samples a local edit action conditioned on the current skill state and source evidence at each step. Applying these actions progressively produces the skill that conditions the fixed worker on downstream tasks.
Group Relative Policy Optimization. Group Relative Policy Optimization (GRPO) (DeepSeek-AI et al., 2025) is used to optimize the skill generator. For an input , GRPO samples a group of edit actions from the old policy and assigns each action a scalar reward . The group-relative advantage is . The policy is then optimized with the clipped objective
(1)
where , is the reference policy, and are hyperparameters.
Skill Generation Formalization. For each task family , let denote a conceptual high-quality teacher behavior distribution. Let denote a sequence of evidence units (e.g., documents, task contexts, or execution traces) induced by a source query from the same task family, i.e., . Given an initial skill state , the skill-editing policy samples local edit actions whose progressive application induces a distribution over the final skill . The value of a generated skill is defined by how it changes the behavior distribution of the fixed worker agent on a held-out target query from the same task family, where typically . The ideal objective of skill generation is
(2)
This objective formalizes skill generation as learning to construct a skill from source evidence induced by so that the fixed worker moves toward high-quality behavior on a held-out target query . Since the teacher behavior distribution is unavailable, this ideal objective cannot be directly optimized and must be approximated through an indirect but related reward signal.
4 Method: Skill-
To approximate the ideal objective in Eq. 2, we develop Skill- as a trainable framework. It is a for progressive skill generation framework. In the following subsections, we will introduce how decompose skill generation into local edits in Section 4.1, introduce rollback reward for edit-level credit assignment in Section 4.2, and present the training implementation in Section 4.3. We illustrate the overall framework in Figure 1.
4.1 Progressive Skill Generation
Ideally, a generator should read all source evidence induced by and produce a complete skill in one shot, regardless of the generation strategy. In practice, the source evidence often exceeds the context window of an LLM because individual units are too long or the sequence contains too many units, causing key information to be missed. Moreover, one-shot skill generation entangles several operations, including abstraction, denoising, conflict resolution, and compression, which makes credit assignment weak and training difficult. To address this issue, we construct the skill through a sequence of local edit decisions rather than one-shot generation. Starting from an initial skill , the generator reads evidence sequentially and updates the current skill one step at a time, i.e.,
(3)
Here, is a local edit conditioned on the current skill state and the evidence , and applies this edit to obtain the next skill state . After steps, the final skill is a sample from the induced distribution in Eq. 2.
This progressive formulation covers both document-to-skill and experience-to-skill settings and reduces skill generation to a unified local decision problem, i.e., how should each evidence unit be used to edit the current skill so that the fixed worker performs better on held-out target queries? We use a rollback reward to provide supervision for these edit decisions.
Figure 1: An overview of Skill-. Left: during inference, Skill- reads evidence sequentially and applies a sequence of local edit actions to progressively generate the skill from the initial state to the final skill . Right: during training, the skill generator samples a group of candidate actions from the current skill state, constructs the corresponding edited skills, and evaluates them on the same evidence-related anchored query with a benchmark-specific verifier; the resulting rollback rewards are then used for the GRPO update.
4.2 Action Space and Rollback Reward
We define the edit action space . Each sampled action edits a single skill artifact. Create adds missing rules or procedures; Update fixes incomplete or inaccurate rules; Merge consolidates overlapping content; Prune removes misleading, overly specific, or redundant content; and Noop leaves the skill unchanged when the current evidence provides no useful new information.
The remaining challenge is edit-level reward assignment. Directly rewarding an edit by rolling out the worker agent with the edited skill is insufficient, since a correct downstream answer may come from the capability of the worker agent or from an easy query, rather than from the edit itself. We therefore introduce rollback reward, which assigns credit by comparing the old and edited skills on the same evidence-related anchored query. For an edit decision at step , we attach an anchored query selected from queries associated with the same evidence source, together with a benchmark-specific verifier . After applying to obtain , the fixed worker answers the same anchored query under the original and edited skills:
(4)
The verifier then assigns scalar feedback to the two outcomes, i.e., and . In our main setup, CL-Bench uses official task-specific rubrics, while SpreadsheetBench and tau2-bench use direct benchmark-side environment feedback on the same anchored evaluation query. In GRPO training, rollback rewards are assigned within each sampled action group. For a sampled candidate , we write its edited verifier output as . Formally,
(5)
This local reward is used during RL for edit-level credit assignment, while full-task execution remains the evaluation target. Appendix A shows that expected rollback reward preserves ideal-preference rankings under a calibrated verifier and gives an exact success-probability ordering for a binary verifier, providing a local justification for group-relative optimization.
4.3 Training Implementation
媒体内容 · 前往原文查看
system: Skill generation action policy.
user:
## Current SKILL.md
current skill content
## Evidence
serialized source evidence
assistant:
evidence-grounded diagnosis and edit rationale
{"action": "…", …}
Table 1: Training template for Skill-. The policy writes a reasoning trace and one structured edit action for the current skill and evidence.
媒体内容 · 前往原文查看
Algorithm 1 Reinforcement Learning with Rollback Reward for Skill-
1: Inputs: local editing-state dataset , skill generator , fixed worker , group size .
2: for each local editing state do
3: Evaluate the control skill on the anchored query and obtain .
4: Sample candidate edit actions from .
5: for do
6: Apply to the current skill and obtain .
7: Evaluate the edited skill on the same anchored query and obtain .
8: end for
9: Compute the rollback rewards with Eq. 5, including the Noop fallback rule.
10: Update on the candidate group using GRPO as defined in Eq. 1.
11: end for
12: Return: Optimized skill generator .
We train as a structured skill-editing policy rather than a task-solving policy by initializing it from instruction-tuned Qwen3-8B (Yang et al., 2025), warming it up with supervised skill-editing data, and then optimizing it with GRPO. Each training unit contains the current skill , a serialized evidence batch , and an anchored query with its verifier for rollback evaluation. As shown in Table 1, the policy outputs a reasoning trace and one structured edit action for this local editing state. The input construction is shared across both evidence types. For document-to-skill, short contexts are edited directly, while long contexts are decomposed into ordered natural segments and processed progressively. For experience-to-skill, evidence is built from GPT-4o trajectories sampled on the same training split used for Skill-, then serialized into a compact standardized view that preserves the task identity, key steps, and environment feedback. In the main setting, each experience batch contains up to traces, each trace contributes at most steps and characters, and the prompt budget is capped at tokens. This shared serialization keeps the local editing state within context limits while exposing comparable evidence structure across benchmarks.
During the SFT stage, the warm-up data is synthesized by DeepSeek-V4-Pro (DeepSeek-AI et al., 2026) from the training splits of CL-Bench, SpreadsheetBench, and tau2-bench. We keep only filtered edit trajectories that follow the shared + format, so that the policy learns the action syntax, local edit structure, and basic evidence-grounded editing behavior under the same interface used later in RL. The resulting checkpoint is then used to initialize GRPO.
After warm-up, we optimize the policy with GRPO while keeping the worker agent fixed as GPT-4o (OpenAI, 2024). For a local editing state , the old policy samples a group of candidate actions . The current skill is evaluated once on the anchored query to obtain the control score , and each edited skill is then evaluated with the same worker and verifier to obtain . For CL-Bench, is a GPT-5.5 (OpenAI, 2026) rubric judge instantiated with the official task-specific evaluation rules. For SpreadsheetBench and tau2-bench, is the direct benchmark-side environment feedback returned by the same anchored execution interface. These scores are converted into rollback rewards by Eq. 5. This design trains the policy on local edit states rather than directly optimizing a full generation trajectory, which keeps reward assignment aligned with the progressive formulation in Section 4.1.
For training, SFT uses GPUs, a training batch size of , a micro-batch size of per GPU, a maximum sequence length of , a learning rate of , a weight decay of , a warmup ratio of , cosine scheduling, and epochs. GRPO uses a group size of , a training batch size of , an actor learning rate of , a rollout temperature of , an entropy coefficient of , a maximum prompt length of , a maximum response length of , and a maximum model length of , with KL regularization disabled. The overall procedure is shown in Algorithm 1. More detailed implementation details, including exact evidence preprocessing, anchored-query construction, and verifier protocols, are given in Appendix B.1.
5 Experiments
5.1 Evaluation Setup
Benchmarks and Baselines. We evaluate Skill- in both document- and experience-to-skill settings. For document-to-skill, we use CL-Bench (Dou et al., 2026). Training uses Rule System Application and Procedural Task Execution as source categories, and evaluation reports held-out tasks from these two categories together with the unseen categories Domain Knowledge Reasoning and Empirical Discovery & Simulation. Since CL-Bench is evaluated with LLM-as-judge under task-specific rubrics, we use GPT-5.5 as the CL-Bench judge for all reported results. We compare against No Skill, Anthropic Skill-Creator (Anthropic, 2026), Progressive Prompt Skill, Ctx2Skill (Si et al., 2026), and AutoSkill (Yang et al., 2026). For experience-to-skill, we use SpreadsheetBench (Ma et al., 2024) and tau2-bench (Barres et al., 2026). SpreadsheetBench tests execution-based spreadsheet manipulation, while tau2-bench tests workflow skill reuse across Airline, Retail, and Telecom. The experience baselines are No Skill, Anthropic Skill-Creator, Progressive Prompt Skill, ExpeL (Zhao et al., 2024), Agent Workflow Memory (AWM) (Wang et al., 2025c), Trace2Skill (Ni et al., 2026), SkillX (Wang et al., 2026a), and SkillPro (Mi et al., 2026). Baseline details are provided in Appendix B.2.
Evaluation. All methods follow the same source-evidence and held-out-task protocol. For document-to-skill, every method receives the same source contexts and is evaluated on the same held-out tasks after injecting the generated skill into the worker. For experience-to-skill, the source trajectories are sampled once by GPT-4o on the benchmark training split, and this source split is exactly the trajectory training split used by Skill-; all automatic baselines receive the same trajectories, tool observations, and execution feedback for skill construction. We report same-worker evaluation with GPT-4o and cross-worker transfer to Claude-Sonnet-4.5 (Anthropic, 2025). Note that Claude-Sonnet-4.5 is used only as a downstream worker that consumes the same skills generated from the GPT-4o trajectory pool. We report pass rates on CL-Bench, SpreadsheetBench, and tau2-bench. Detailed benchmark splits, worker settings, and leakage-prevention rules are given in Appendix C.
5.2 Main Results
媒体内容 · 前往原文查看
Table 2: Document-to-skill results on CL-Bench. Avg. is the unweighted average over the four CL-Bench categories. All numbers are pass rates (%), and indicates that higher is better. Skills are generated from source contexts and then reused on held-out tasks under both worker backbones; Claude-Sonnet-4.5 is used only as a transfer worker. Best and second-best results within each backbone block are in bold and underlined.
Method CL-Bench Rule System Application Procedural Task Execution Domain Knowledge Reasoning Empirical Discovery & Simulation Avg. Backbone: GPT-4o No Skill 21.82 4.30 5.13 3.02 8.57 Anthropic Skill-Creator 14.55 5.38 4.07 4.02 7.01 Progressive Prompt Skill 16.36 4.30 3.77 4.02 7.11 AutoSkill 15.45 3.23 3.77 3.02 6.37 Ctx2Skill 15.82 4.30 3.02 5.13 7.07 Skill- 20.91 9.68 5.88 5.03 10.38 Backbone: Claude-Sonnet-4.5 No Skill 17.27 2.15 4.83 6.53 7.70 Anthropic Skill-Creator 15.45 1.08 4.68 8.04 7.31 Progressive Prompt Skill 18.18 2.15 4.68 6.53 7.89 AutoSkill 19.09 1.08 5.88 7.54 8.40 Ctx2Skill 18.18 2.15 7.54 4.98 8.21 Skill- 19.09 3.22 7.84 8.04 9.55
Document-to-skill. Table 2 shows that Skill- achieves stable context-compression and skill-construction performance on CL-Bench. Under GPT-4o, the largest gain appears on Procedural Task Execution, where Skill- improves from without skills to , substantially exceeding Anthropic Skill-Creator (), AutoSkill (), and Ctx2Skill (). This suggests that Skill- is not merely shortening long contexts but turning procedural constraints into executable skills. On Domain Knowledge Reasoning and Empirical Discovery & Simulation, it is best or near-best, while on Rule System Application it remains close to the No Skill baseline, indicating that the generated skills preserve information useful for both procedural and reasoning-heavy tasks. Under Claude-Sonnet-4.5, Skill- is best or tied-best on all four categories and achieves the highest CL-Bench average. Since Claude already has a relatively strong No Skill baseline, these gains indicate that the generated skills not only fit the backbone used for skill construction but also provide reusable structured knowledge that transfers to another strong worker.
媒体内容 · 前往原文查看
Table 3: Experience-to-skill results on SpreadsheetBench and tau2-bench. tau2-bench Avg. is the unweighted average over Airline, Retail, and Telecom. All numbers are pass rates (%), and indicates that higher is better. Skills are generated from the shared GPT-4o trajectory pool and then reused on held-out tasks under both worker backbones; Claude-Sonnet-4.5 is used only as a transfer worker. Best and second-best results within each backbone block are in bold and underlined.
Method SpreadsheetBench tau2-bench Airline Retail Telecom Avg. Backbone: GPT-4o No Skill 18.00 40.00 47.50 12.50 33.33 Anthropic Skill-Creator 26.00 55.00 60.00 7.50 40.83 Progressive Prompt Skill 18.00 45.00 60.00 10.00 38.33 ExpeL 18.50 55.00 70.00 12.50 45.83 AWM 20.00 40.00 72.50 15.00 42.50 Trace2Skill 19.00 45.00 67.50 12.50 41.67 SkillX 18.50 50.00 80.00 8.00 46.00 SkillPro 15.50 55.00 72.50 20.00 49.17 Skill- 27.50 65.00 80.00 22.50 55.83 Backbone: Claude-Sonnet-4.5 No Skill 26.00 65.00 85.00 40.00 63.33 Anthropic Skill-Creator 20.00 80.00 80.00 37.50 65.83 Progressive Prompt Skill 13.50 55.00 87.50 42.50 61.67 ExpeL 31.00 60.00 85.50 40.00 61.83 AWM 25.00 65.00 85.50 32.50 61.00 Trace2Skill 20.00 70.00 86.00 32.50 62.83 SkillX 31.50 75.00 85.00 35.00 65.00 SkillPro 20.50 60.00 87.00 45.00 64.00 Skill- 31.50 80.00 86.00 45.00 70.33
Experience-to-skill. Table 3 shows that the advantage of Skill- is even clearer in experience-to-skill. Under GPT-4o, it is best or tied-best on every reported metric, improving SpreadsheetBench from to , Airline from to , and Telecom from to , while matching the best Retail result of and achieving a tau2-bench average of . These gains suggest that Skill- learns reusable execution strategies rather than merely recording experience or writing static rules. Under Claude-Sonnet-4.5, Skill- remains best or tied-best on SpreadsheetBench, Airline, Telecom, and the tau2-bench average, and stays close on Retail, showing that the learned skills transfer across workers even though Claude only consumes GPT-4o-generated skills.
Necessity of Training. To test whether training is necessary, we compare against two strong prompt baselines, i.e., Anthropic Skill-Creator and Progressive Prompt Skill. They can be competitive on individual columns, reaching on SpreadsheetBench and on Airline under GPT-4o, and on Retail under Claude-Sonnet-4.5, but their gains are unstable across tasks and workers. Anthropic drops to on Telecom, below the No Skill baseline of , and Progressive drops to on SpreadsheetBench, far below the No Skill baseline of . Since Progressive already performs iterative refinement, the more consistent gains of Skill- indicate that training a skill-editing policy is necessary rather than repeatedly applying a fixed prompt.
Generalization Across Evidence Sources and Workers. Taken together, the two main tables support the central claim of this paper. Most baselines are evidence-source specific. AutoSkill and Ctx2Skill are designed for document evidence, whereas Trace2Skill, ExpeL, AWM, SkillX, and SkillPro are designed for experience evidence. In contrast, Skill- uses one progressive skill editing framework across both sources and remains strong in both settings. The cross-worker results provide a second form of generalization evidence. Gains are larger under GPT-4o, suggesting that the generated skills effectively supplement the backbone used for skill construction. At the same time, the same skills remain useful under Claude-Sonnet-4.5, which indicates that Skill- is learning externally usable task-solving knowledge rather than a worker-specific prompt shortcut.
5.3 Ablation Study and Training Dynamics
We study four ablations under the same evaluation protocol. “SFT only” removes RL and keeps only the supervised warm start. “w/o rollback reward” replaces rollback reward with direct verifier reward. “w/o Merge/Prune” removes the structural consolidation and deletion actions from the edit space. “w/o Noop” removes the abstention action and forces every step to modify the current skill.
媒体内容 · 前往原文查看
Table 4: Ablation study. CL-Bench Avg. averages the four CL-Bench categories, and tau2-bench Avg. averages Airline, Retail, and Telecom. All numbers are pass rates (%).
Variant CL-Bench Avg. SpreadsheetBench tau2-bench Avg.
Skill- 10.38 27.50 55.83
SFT only 3.46 15.50 44.17
w/o rollback reward 3.68 17.00 46.67
w/o Merge/Prune 4.74 20.00 39.17
w/o Noop 9.55 22.00 53.33
Figure 2: Training dynamics and rollout action distribution under the main ablations. Left: training reward over RL steps for Skill- and its RL ablations. Right: action distribution accumulated over rollout steps. “SFT only” is omitted because it has no RL rollout trajectory.
Ablation Performance. Table 4 shows that the full model is strongest on all three benchmarks, with gains of , , and over “SFT only”, which confirms that the benefit does not come from learning the output format alone. Without rollback reward, performance stays close to “SFT only” at , , and , which makes rollback reward the key signal tying a local edit to downstream improvement. Removing Merge/Prune also causes a clear drop, especially on the tau2-bench average (), showing that skill construction requires explicit consolidation and deletion rather than continual accumulation alone. Removing Noop still leaves a relatively strong model at , , and , which rules out a trivial no-edit explanation, but the full model remains consistently better, indicating that Noop is useful as a calibrated action when the current skill is already adequate.
Training Dynamics and Edit Behavior. Figure 2 explains the ablation results from a complementary perspective. The full model does not start with the highest reward, but improves gradually over training, which suggests that it is learning a better editing strategy rather than exploiting a fixed shortcut. Its action distribution is also balanced. Specifically, Create is the dominant action, but Update, Merge, Prune, and Noop all remain active, indicating a mixed policy that both adds and maintains the skill. By contrast, removing rollback reward produces a noisier, flatter reward trajectory and an action distribution dominated by Noop, consistent with a conservative no-edit policy under weak credit assignment. Removing Merge/Prune yields short-term reward gains but weaker later-stage behavior, which matches the view that creation alone cannot prevent redundant or conflicting skills from accumulating. Removing Noop still preserves active editing and a relatively strong reward curve, confirming that Skill- does not rely on abstention as a shortcut, but its lower downstream scores show that forced edits still introduce avoidable noise.
5.4 Analysis on Evidence
Evidence Order. Table 5 first studies evidence order while fixing the batch size to , so only the ordering changes and the evidence pool remains the same. Here, source order preserves the original source sequence, shuffled order randomly permutes the same evidence units, and reverse order flips that sequence. The effect of order is visible but not decisive. These results indicate that Skill- is fairly robust to evidence order and does not simply rely on one fixed trajectory arrangement. At the same time, the weaker shuffled result on tau2 suggests that local continuity can still help progressive editing when adjacent evidence units refine related workflow structure.
媒体内容 · 前往原文查看
Table 5: Analysis of evidence order and batch size. tau2-bench Avg. averages Airline, Retail, and Telecom. All numbers are pass rates (%).
Variant SpreadsheetBench tau2-bench Avg.
Evidence order
Source order 27.50 55.83
Shuffled order 28.00 51.67
Reverse order 26.00 57.50
Evidence batch size
1 evidence unit / step 22.00 47.50
2 evidence units / step 23.50 53.33
4 evidence units / step 27.50 55.83
8 evidence units / step 20.00 48.33
Evidence Batch Size. The same table then varies batch size while fixing the order to source order. Here the effect is much stronger. The best setting uses evidence units per step, while both smaller and larger batches degrade performance. This pattern shows that progressive skill generation depends on evidence granularity rather than on simply exposing as much evidence as possible at once. When the batch is too small, the editor becomes myopic and tends to overreact to local failures or trajectory-specific details, which makes the skill easier to fragment into narrow updates. When the batch is too large, the editor becomes overloaded because multiple patterns, failures, and candidate edits are mixed together, weakening edit-level focus and making credit assignment less clear. A moderate batch gives the best trade-off because it provides enough shared pattern support for abstraction while still keeping the editing target local and controllable. This also clarifies that Skill- needs sufficient but not excessive evidence at each step to support reusable abstraction during progressive generation.
6 Conclusion and Future Work
We present Skill-, a reinforcement learning framework that formulates skill generation as a progressive editing process over a single evolving skill artifact and trains local edit decisions with rollback reward on held-out evaluation anchors. In this way, Skill- optimizes skill generation based on how a skill edit changes downstream worker behavior rather than on textual plausibility alone. Experiments on CL-Bench, SpreadsheetBench, and tau2-bench show that Skill- produces more effective skills than prompting-based and pipeline-style baselines in both document-to-skill and experience-to-skill settings, and that these gains transfer across worker backbones. Ablation and training-dynamics analyses further show that the benefit comes from rollback-grounded learning together with the full edit space, while additional analysis shows that progressive generation is robust to evidence reordering but depends more strongly on using a moderate evidence batch size. A current limitation is that the reward and verifier interface is still benchmark-dependent, and the current skill representation is text-based. Future work can extend it to stronger and more general verifiers, richer multimodal skill formats, and longer-horizon training signals that go beyond local rollback comparisons.
References
Anthropic (2025) Anthropic. Claude Skills. https://claude.com/blog/skills, October 2025. Accessed: 2026-05-27.
Anthropic (2025) Anthropic. Introducing claude sonnet 4.5, 2025. URL https://www.anthropic.com/news/claude-sonnet-4-5.
Anthropic (2026) Anthropic. Skill creator. https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md, March 2026. Accessed: 2026-05-27.
Barres et al. (2026) Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. -bench: Evaluating conversational agents in a dual-control environment. In Proceedings of the 43rd International Conference on Machine Learning, Seoul, South Korea, 2026.
Cheng et al. (2025) Ming Cheng, Jinhua Ouyang, Shulin Yu, Ruochen Yan, Yuchen Luo, Zihan Liu, and Enhong Chen. Agent-r1: Training powerful llm agents with end-to-end reinforcement learning. arXiv preprint arXiv:2511.14460, 2025.
DeepSeek-AI et al. (2025) DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025.
DeepSeek-AI et al. (2026) DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Donghao Li, Dongjie Ji, Erhang Li, Fang Wei, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei, Haotian Yuan, Haowei Zhang, Haowen Luo, Haoyu Chen, Haozhe Ji, Hengqing Zhang, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J Yang, JQ Zhu, Jia Luo, Jia Song, Jia Yu, Jialiang Huang, Jialu Cai, Jian Liang, Jiangting Zhou, Jiasheng Ye, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jieyu Yang, Jin Chen, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jingzi Zhou, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junmin Zheng, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Leyi Xia, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao Luo, Linwang Ma, Linyan Zhu, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, MS Di, MY Xu, Max Mei, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Mingxu Zhou, Minmin Han, Ning Wang, Panpan Huang, Panpan Wang, Peixin Cong, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, Ruyi Chen, SH Liu, Shanghao Lu, Shangmian Sun, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoqing Wu, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W Zhang, WL Xiao, Wangding Zeng, Wei An, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Yang, Wenlve Huang, Wenqing Hou, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen Wang, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyu Zhang, Xu Chen, Xuanyu Wang, Xuecheng Su, Xueyin Chen, Xuheng Lin, Xuwei Fu, YC Yan, YQ Wang, YW Ma, Yanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Xu, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Shao, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang Xiong, Yiling Ma, Ying He, Ying Tang, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, YuKun Li, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuehan Yang, Yuer Xu, Yuhan Wu, Yuhao Meng, Yuheng Zou, Yukun Zha, Yunfan Xiong, Yupeng Chen, Yuping Lin, Yuqian Cao, Yuqian Wang, Yushun Zhang, Yuting Yan, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhen Huang, ZF Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zekai Zhang, Zhangli Sha, Zhe Fu, Zhe Ju, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zheren Gao, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhongyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihua Qu, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Ziyi Wan, Zizheng Pan, and Zongqing Yao. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.
Dou et al. (2026) Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, et al. Cl-bench: A benchmark for context learning. arXiv preprint arXiv:2602.03587, 2026.
Huang et al. (2024) Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716, 2024.
Huang et al. (2026) Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Xue Yang, Dongdong Chen, Xiaoqing Zheng, and Chong Luo. From raw experience to skill consumption: A systematic study of model-generated agent skills. arXiv preprint arXiv:2605.23899, 2026.
Jin et al. (2025) Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
Li et al. (2026) Xiangyi Li, Yimin Liu, Wenbo Chen, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Binxu Li, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jiachen Li, Songlin Li, Yijiang Li, Yueqian Lin, Xinyi Liu, Xuanqing Liu, Haoran Lyu, Ze Ma, Kaixin Li, Runhui Wang, Tianyu Wang, Wengao Ye, Yue Zhang, Hanwen Xing, Yiqi Xue, Steven Dillmann, and Han-chung Lee. Skillsbench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026.
Li et al. (2025) Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025.
Liu et al. (2026) Dawei Liu, Zongxia Li, Hongyang Du, Xiyang Wu, Shihang Gui, Yongbei Kuang, and Lichao Sun. Graph of skills: Dependency-aware structural retrieval for massive agent skills. arXiv preprint arXiv:2604.05333, 2026.
Luo et al. (2025) Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S. Yu, and Ming Zhang. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460, 2025.
Ma et al. (2024) Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. Spreadsheetbench: Towards challenging real world spreadsheet manipulation. In Advances in Neural Information Processing Systems 37, pp. 94871–94908, Vancouver, Canada, 2024.
Mi et al. (2026) Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, and Jun Wang. Skill-pro: Learning reusable skills from experience via non-parametric ppo for llm agents. arXiv preprint arXiv:2602.01869, 2026.
Ni et al. (2026) Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026.
OpenAI (2024) OpenAI. Hello GPT-4o, 2024. URL https://openai.com/index/hello-gpt-4o/.
OpenAI (2026) OpenAI. Introducing GPT‑5.5, 2026. URL https://openai.com/index/introducing-gpt-5-5/.
Ouyang et al. (2026) Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, et al. Skillos: Learning skill curation for self-evolving agents. arXiv preprint arXiv:2605.06614, 2026.
Plaat et al. (2026) Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. Multi-step reasoning with large language models, a survey. ACM Computing Surveys, 58(6):160:1–160:35, 2026.
Qi et al. (2024) Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024.
Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Shen et al. (2025) Junhao Shen, Haiteng Zhao, Yuzhe Gu, Songyang Gao, Kuikun Liu, Haian Huang, Jianfei Gao, Dahua Lin, Wenwei Zhang, and Kai Chen. Semi-off-policy reinforcement learning for vision-language slow-thinking reasoning. In Advances in Neural Information Processing Systems 38, San Diego, CA, 2025.
Shen et al. (2026) Junhao Shen, Teng Zhang, Xiaoyan Zhao, and Hong Cheng. Dynamic skill lifecycle management for agentic reinforcement learning. arXiv preprint arXiv:2605.10923, 2026.
Shen (2024) Zhuocheng Shen. Llm with tools: A survey. arXiv preprint arXiv:2409.18807, 2024.
Si et al. (2026) Shuzheng Si, Haozhe Zhao, Yu Lei, Qingyi Wang, Dingwei Chen, Zhitong Wang, Zhenhailong Wang, Kangyang Luo, Zheng Wang, Gang Chen, et al. From context to skills: Can language models learn from context skillfully? arXiv preprint arXiv:2604.27660, 2026.
Singh et al. (2025) Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning. arXiv preprint arXiv:2505.01441, 2025.
Wang et al. (2026a) Chenxi Wang, Zhuoyun Yu, Xin Xie, Wuguannan Yao, Runnan Fang, Shuofei Qiao, Kexin Cao, Guozhou Zheng, Xiang Qi, Peng Zhang, and Shumin Deng. Skillx: Automatically constructing skill knowledge bases for agents. arXiv preprint arXiv:2604.04804, 2026a.
Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
Wang et al. (2026b) Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, et al. Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents. arXiv preprint arXiv:2604.10674, 2026b.
Wang et al. (2025a) Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong. Reinforcement learning for self-improving agent with skill library. arXiv preprint arXiv:2512.17102, 2025a.
Wang et al. (2024) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
Wang et al. (2025b) Yu Wang, Ryuichi Takanobu, Zhiqi Liang, Yuzhen Mao, Yuanzhe Hu, Julian McAuley, and Xiaojian Wu. Mem-: Learning memory construction via reinforcement learning. arXiv preprint arXiv:2509.25911, 2025b.
Wang et al. (2026c) Zhaoyang Wang, Qianhui Wu, Xuchao Zhang, Chaoyun Zhang, Wenlin Yao, Fazle Elahi Faisal, Baolin Peng, Si Qin, Suman Nath, Qingwei Lin, et al. Webxskill: Skill learning for autonomous web agents. arXiv preprint arXiv:2604.13318, 2026c.
Wang et al. (2026d) Zimu Wang, Yuling Shi, Mengfan Li, Zijun Liu, Jie M Zhang, Chengcheng Wan, and Xiaodong Gu. Effiskill: Agent skill based automated code efficiency optimization. arXiv preprint arXiv:2603.27850, 2026d.
Wang et al. (2025c) Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada, 2025c.
Xia et al. (2026) Peng Xia, Jing Chen, Hao Wang, Jia Liu, Kai Zeng, Yixiao Wang, and Haoran Yao. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234, 2026.
Xu & Nie (2025) Jingkai Xu and Xiangli Nie. Speci: Skill prompts based hierarchical continual imitation learning for robot manipulation. IEEE Transactions on Cognitive and Developmental Systems, 2025.
Yan et al. (2025) Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z Pan, et al. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv preprint arXiv:2508.19828, 2025.
Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Yang et al. (2026) Yutao Yang, Junsong Li, Qianjun Pan, Bihao Zhan, Yuxuan Cai, Lin Du, Jie Zhou, Kai Chen, Qin Chen, Xin Li, Bo Zhang, and Liang He. Autoskill: Experience-driven lifelong learning via skill self-evolution. arXiv preprint arXiv:2603.01145, 2026.
Yu et al. (2025) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
Zeng et al. (2026) Kun Zeng, Yu Huo, Siyu Zhang, Zi Ye, Yuecheng Zhuo, Haoyue Liu, Yuquan Lu, Junhao Wen, and Xiaoying Tang. Group of skills: Group-structured skill retrieval for agent skill libraries. arXiv preprint arXiv:2605.06978, 2026.
Zhao et al. (2024) Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pp. 19632–19642, Vancouver, Canada, 2024.
Zhao et al. (2026) Haiteng Zhao, Junhao Shen, Yiming Zhang, Songyang Gao, Kuikun Liu, Tianyou Ma, Fan Zheng, Dahua Lin, Wenwei Zhang, and Kai Chen. Achieving olympia-level geometry large language model agent via complexity boosting reinforcement learning. In Proceedings of the 14th International Conference on Learning Representations, Rio de Janeiro, Brazil, 2026.
Zheng et al. (2026) YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu, JiHuai Zhu, Yong Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, et al. Skillrouter: Skill routing for llm agents at scale. arXiv preprint arXiv:2603.22455, 2026.
Zhou et al. (2026) Yifan Zhou, Zhentao Zhang, Ziming Cheng, Shuo Zhang, Qizhen Lan, Zhangquan Chen, Zhi Yang, Ronghao Chen, Huacan Wang, Sen Hu, et al. Skillgenbench: Benchmarking skill generation pipelines for llm agents. arXiv preprint arXiv:2605.18693, 2026.
The appendix is organized as follows: Appendix A presents the theoretical analysis of Skill-. Appendix B describes the detailed implementation of Skill- and baselines. Appendix C describes the detailed experimental setup. Appendix D presents all prompts used in training, evaluation, and skill expansion. Appendix E discusses the limitations of Skill-.
Appendix A Theoretical Analysis
This section analyzes the rollback reward used in Section 4.2. We first establish its exact pairwise interpretation and then give sufficient conditions under which it preserves ideal-preference and binary-success ordering. We finally characterize the group-dependent Noop reward. The analysis concerns the local ranking signal used by GRPO.
Fix a local editing state, including the current skill, anchored query, worker, verifier, and evaluation protocol, and condition on a sampled group of valid actions. Let be the verifier output under the current skill and the output under the skill produced by a non-Noop action . Following Eq. 5, its reward is
(A1)
Assumption A.1.
The control and edited branches use the same anchored query, worker policy, verifier, environment configuration, and sampling protocol. Conditional on the local editing state and candidate skills, the branch rollouts are mutually independent, and the skill condition is the only systematic difference between the control branch and each edited branch.
Lemma A.2.
For every non-Noop candidate , the expected rollback reward is
(A2)
Thus, one rollback comparison is a Bernoulli sample whose mean is the verifier pairwise win probability of the edited skill against the control skill.
Proof. Equation A1 defines as the indicator of the event . Taking its expectation gives
(A3)
This completes the proof. ∎
Let denote the ideal pairwise judgment on the anchored query, where means that the edited rollout better satisfies the task criterion than the control rollout. Let denote the corresponding ideal-preference probability. We use the following calibration condition to relate the implemented verifier comparison to this latent judgment.
Assumption A.3.
For a fixed local editing state, the verifier has candidate-independent false-positive and false-negative rates such that
(A4)
for every valid non-Noop candidate, where . A verifier that exactly implements the ideal preference is the special case .
Lemma A.4.
Under Assumption A.1 and Assumption A.3, the expected rollback reward satisfies
(A5)
Consequently, for any two non-Noop candidates,
(A6)
Proof. Assumption A.1 ensures that the verifier and ideal judgments refer to the same local comparison. Since is binary, . By the law of total probability and Assumption A.3,
(A7)
Because , Eq. A5 is strictly increasing in , which proves Eq. A6. This completes the proof. ∎
Lemma A.5.
Suppose the verifier is the exact binary success indicator. Let be the worker success probability under the control skill and the success probability under the skill produced by . Under Assumption A.1,
(A8)
Consequently, if , then for any two candidate edits,
(A9)
Proof. For an exact binary verifier, precisely when the edited rollout succeeds and the control rollout fails. Assumption A.1 therefore gives
(A10)
(A11)
The common factor is positive when , which proves Eq. A9. This completes the proof. ∎
Lemma A.6.
Let be the set of valid non-Noop candidates in a sampled group. If is nonempty, then for an arbitrary scalar verifier, the expected Noop reward is
(A12)
Under Assumption A.1, let denote the verifier-score CDF of candidate . Then
(A13)
In the exact binary setting of Lemma A.5, this expression reduces to
(A14)
If is empty, Eq. 5 assigns reward one to Noop deterministically.
Proof. Equation 5 rewards Noop precisely when for every , which gives Eq. A12. Conditional on , Assumption A.1 gives
(A15)
Taking the expectation over the control score proves Eq. A13. For an exact binary verifier, a successful control cannot be exceeded and contributes probability . When the control fails, Noop is rewarded only if every edited rollout also fails, which contributes . This proves Eq. A14 and completes the proof. ∎
Assumption A.1 makes the rollback comparison consistent across the control and edited branches. Lemma A.2 identifies its exact expected quantity, while Assumption A.3 and Lemma A.4 give sufficient conditions for this quantity to preserve ideal-preference ordering. Lemma A.5 provides an exact specialization for binary task success, and Lemma A.6 separately characterizes Noop as a group-level fallback rather than an estimator of edit quality. Together, these results justify rollback reward for its role in local group-relative ranking. Ties and degradations both receive zero reward, so the reward is not a signed estimate of score improvement, and one comparison remains a noisy sample of the corresponding win probability. Independent branches are sufficient for the expectations above, while repeated comparisons can reduce variance without changing the estimand.
Appendix B Implementation Details
B.1 Skill- Setup
Policy and Action Format. Skill- uses the shared skill-generation interface in Table 1 for both document-to-skill and experience-to-skill. The policy receives the current SKILL.md and serialized evidence, then emits a short rationale and exactly one structured action. Table A1 specifies the required fields and editing semantics of the five actions. The action applier normalizes headings and applies the sampled edit to the current Markdown skill without rewriting it into another action.
媒体内容 · 前往原文查看
Action Required fields Intended effect
Create sections add one or more missing reusable sections
Update target_title, new_content replace one existing section as a whole
Merge source_titles, merged_title, merged_content consolidate overlapping sections
Prune target_title remove harmful or redundant content
Noop — keep the current skill unchanged
Table A1: Structured action interface for Skill-. Create supports multi-section creation; Update, Merge, and Prune operate on existing sections; and Noop preserves the current artifact.
Evidence and Anchored Queries. The same edit policy is trained across both evidence regimes, while benchmark-specific logic appears only in evidence serialization and reward evaluation. For CL-Bench, evidence is serialized from training contexts. Short contexts are edited directly, while long contexts are processed as an ordered sequence of natural context segments; after each segment, the current skill is updated and passed to the next step. Each RL state is paired with a training query whose task reference belongs to the same source context. We use the task reference attached to the state without filtering for an initially failed response, and GPT-5.5 scores the control and edited answers under the corresponding official rubric. The anchored query and rubric are reserved for rollback evaluation rather than introduced as additional fields in the editing-policy interface. For SpreadsheetBench and tau2-bench, evidence is serialized from training trajectories, tool observations, final outcomes, and feedback. We construct the anchor pool from unsuccessful or partially successful trajectories in the same source group, identified by an unsuccessful final status or a score below one, and shuffle this pool before selecting anchors. The selected anchor is excluded from the evidence traces in that local state and is used only to compare the control and edited skills through the benchmark environment. This selection provides local headroom for distinguishing candidate edits while keeping the anchor in the same task family as the evidence. Each experience batch contains four source traces in their original order, each trace contributes at most eight steps and characters, and the policy prompt is capped at tokens. No final test query is used as an RL anchor.
Training. Training uses two stages. We initialize the policy from instruction-tuned Qwen3-8B and warm it up before RL. The warm-up data is synthesized by DeepSeek-V4-Pro from the training splits of CL-Bench, SpreadsheetBench, and tau2-bench. After filtering, the SFT corpus contains examples, including from CL-Bench, from SpreadsheetBench, from tau2 Airline, from tau2 Retail, and from tau2 Telecom. The action distribution is Create (), Noop (), Update (), Prune (), and Merge (). Every demonstration follows the shared + format. SFT uses GPUs, a training batch size of , a micro-batch size of per GPU, a maximum sequence length of , a learning rate of , a weight decay of , a warmup ratio of , cosine scheduling, and epochs. The resulting Qwen3-8B checkpoint initializes GRPO. The RL dataset is organized as local editing states rather than full unrolled skill-construction trajectories. Each row binds together the current skill, serialized evidence, one anchored query, and one reward protocol. The final set contains states, comprising CL-Bench states, SpreadsheetBench states, and states from each tau2-bench domain. This mixture keeps RL focused on edit-level credit assignment while preserving source diversity. During RL, the policy samples a group of candidate edits, applies each edit deterministically, and evaluates the resulting candidate skill on the same anchored query used by the control branch. A non-Noop candidate receives positive rollback reward only when its verifier output exceeds the control output, while Noop is rewarded only when no valid sampled edit does so.
Deterministic Quality Gates. Before reward evaluation, candidate edits pass a set of deterministic quality checks. Hard failures include malformed action schemas, missing section targets, duplicate headings, empty post-edit skills after non-Noop actions, and hidden create-or-merge behavior expressed through the wrong action type. A Noop action on an empty current skill is also rejected immediately. Failing candidates receive reward without benchmark evaluation. These gates prevent the policy from exploiting benchmark-specific verifiers through invalid or degenerate edits.
Workers and Verifiers. Rollback reward is always computed using GPT-4o as the fixed worker across RL training queries. Final benchmark reporting is separate: the main paper evaluates generated skills with GPT-4o as the same-worker backbone and Claude-Sonnet-4.5 as the transfer backbone. For CL-Bench, the verifier is GPT-5.5 running the official task-specific rubric protocol. For SpreadsheetBench and tau2-bench, the verifier signal comes directly from the benchmark-side anchored execution feedback returned by the environment on the same query.
B.2 Baseline Setup
Common Settings. All skill-generation baselines receive the same source contexts or GPT-4o trajectory pool used by Skill- and construct one artifact for each source context or experience group. We preserve each method’s native prompt package, artifact representation, update procedure, and selection rule; benchmark wrappers only serialize the shared evidence and materialize the resulting native artifact. Where a method requires an LLM skill writer, we use GPT-4o for a controlled construction backbone. Except for the internal judge defined by Ctx2Skill, baseline generation does not query anchored tasks, benchmark-side verifiers, held-out answers, or test feedback.
No Skill. No Skill applies no skill-generation procedure. It runs the original CL-Bench messages or the official SpreadsheetBench and tau2-bench task interface without external guidance, providing the worker-only reference for each backbone.
Anthropic Skill-Creator. We use the complete Anthropic Skill-Creator instructions as the meta prompt and GPT-4o as the skill generator, with temperature and a maximum output length of tokens. On CL-Bench, the -character context limit reserves characters for prompt overhead, leaving characters for each source chunk. Contexts are divided at paragraph boundaries, with hard splitting for an overlength segment, and chunk-level skills are generated independently and merged once into one context-level SKILL.md. On SpreadsheetBench and tau2-bench, each trajectory retains at most eight steps and characters under the same context limit, and successful trajectories precede the remaining trajectories. Overlength groups are divided into bounded chunks and their partial skills are recursively merged. The final run uses this chunked procedure rather than the optional full-context mode and performs no parameter updates, rollback comparisons, or verifier queries during generation.
Progressive Prompt Skill. This baseline replaces the learned editor with a fixed GPT-4o prompt, using temperature and a maximum output length of tokens. At each step, the prompt receives the current skill and the next bounded evidence chunk and returns a revised full skill. CL-Bench uses the same -character context limit, -character prompt reserve, and -character source-chunk budget as Anthropic Skill-Creator, with contexts processed in their original paragraph order. On SpreadsheetBench and tau2-bench, each trajectory retains at most eight steps and characters under the -character limit, and successful trajectories precede the remaining trajectories. The last revision is used directly as the final skill, without rollback reward, parameter updates, candidate sampling, selection, or a separate merge call.
AutoSkill. We evaluate the native AutoSkill4Doc pipeline on CL-Bench and use GPT-4o as its extraction model. Each source context is exported as one Markdown document and processed by the original document ingestion, extraction, compilation, and version-registration stages. We use the default chunk extraction strategy with a -character context limit, characters per section and extraction chunk, a -character overlap, at most two candidates per unit, and an -token extraction output limit. Other extraction and retry settings remain at their defaults. The compiled artifact is materialized as one context-level SKILL.md; no task query, rubric, or verifier feedback is available during extraction.
Ctx2Skill. We run the native Ctx2Skill pipeline on CL-Bench and preserve its multi-agent self-play, candidate improvement, and selection procedure. Long contexts are divided under its -character context limit before self-play. The full setting uses two workers, five self-play iterations, and five generated tasks per chunk. GPT-4o serves as the challenger, reasoner, proposer, and skill generator, while the method retains the default internal judge and fallback configured by its launcher. The original API calls do not explicitly set temperature or an output-token limit and therefore use the endpoint defaults. The internal judge operates only within Ctx2Skill’s native self-play loop and does not receive CL-Bench task rubrics or held-out benchmark outcomes. Chunk-level outputs are merged into one context-level SKILL.md, with no additional selection outside the native pipeline.
Trace2Skill. We run Trace2Skill with its native trajectory-patch construction and conflict-free consolidation structure, using GPT-4o as the skill writer with temperature and a maximum output length of tokens. Source trajectories are ordered with successful executions first, and each trajectory retains at most eight steps and characters under a -character context limit. Each evidence chunk produces a Markdown patch artifact containing local success and failure lessons, workflow updates, and verification checks. When multiple chunks are required, GPT-4o recursively consolidates their artifacts. The final artifact is one SKILL.md per source group; trajectories are neither pooled across groups nor taken from the test split.
ExpeL. We run ExpeL with its native experience-distillation and retrieval procedure, using GPT-4o as the skill writer with temperature and a maximum output length of tokens. Source trajectories are ordered with successful executions first, and each trajectory retains at most eight steps and characters under a -character context limit. Its prompt converts each evidence chunk into natural-language rules and reusable insights, and recursively merges chunk-level rule sets when needed. At inference, all-mpnet-base-v2 semantic retrieval selects the top two successful source experiences and inserts them with the distilled rules under a -character cap. Retrieval never accesses another domain or a held-out trajectory.
Agent Workflow Memory (AWM). We run AWM with its native offline workflow-induction procedure, using GPT-4o with temperature and a maximum output length of tokens to produce reusable summary workflows. Source trajectories are ordered with successful executions first, and each trajectory retains at most eight steps and characters under a -character context limit. Overlength groups are handled by recursively merging their chunk-level workflow artifacts. At inference, the induced workflow memory is accompanied by the single most relevant successful same-group exemplar retrieved with all-mpnet-base-v2, under a -character cap. This preserves AWM’s workflow-plus-exemplar inference pattern without cross-domain retrieval.
SkillX. We run SkillX with its native three-level library of planning, functional, and atomic skills, using GPT-4o as the skill writer with temperature and a maximum output length of tokens after converting the shared trajectories to its input schema. Source trajectories are ordered with successful executions first, and each trajectory retains at most eight steps and characters under a -character context limit. Structured generations are retried up to three times when JSON validation fails. For overlength groups, chunk-level libraries are merged deterministically by deduplicating entries and retaining at most 25 items in each level. At inference, task-conditioned lexical ranking selects at most ten entries under a -character cap. It prioritizes atomic and tool-centric entries for tau2-bench, while SpreadsheetBench combines one planning entry with functional and atomic entries. The selected view is materialized as promptable guidance while the hierarchical JSON library remains the native artifact.
SkillPro. We run SkillPro with its native option-pool representation, using GPT-4o with temperature and a maximum output length of tokens to generate initiation, policy, and termination fields after converting each source group’s trajectories to its experience schema. Source trajectories are ordered with successful executions first, and each trajectory retains at most eight steps and characters under a -character context limit. Invalid JSON generations are retried up to three times, and chunk-level option pools are merged deterministically with duplicate removal. At inference, task-conditioned lexical ranking chooses at most one option from the corresponding source-group pool under a -character cap. The selected option is rendered as external guidance containing its name, initiation condition, strategy steps, and termination condition; no option may be selected across domains or updated using test outcomes.
Appendix C Evaluation Setup
Benchmarks and Splits. The reported experiments cover one document/context benchmark and two trajectory/experience benchmarks. Table A2 gives the CL-Bench split by context family, and Table A3 summarizes the source/test construction for SpreadsheetBench and tau2-bench.
媒体内容 · 前往原文查看
CL-Bench split Contexts Tasks
Rule System Application (source) 112 456
Procedural Task Execution (source) 80 378
Rule System Application (held-out) 28 110
Procedural Task Execution (held-out) 20 93
Domain Knowledge Reasoning (OOD) 190 663
Empirical Discovery & Simulation (OOD) 70 199
Table A2: CL-Bench split used in this paper. The skill generator is trained only on source contexts from Rule System Application and Procedural Task Execution.
媒体内容 · 前往原文查看
Benchmark split Source tasks Held-out tasks
SpreadsheetBench 200 200
tau2 Airline 30 20
tau2 Retail 74 40
tau2 Telecom 74 40
Table A3: Experience-benchmark splits. SpreadsheetBench uses a stratified 200/200 split of the original 275 cell-level and 125 sheet-level tasks, while tau2-bench follows the official domain-wise split.
CL-Bench Protocol. CL-Bench is used for document-to-skill evaluation. Training uses only Rule System Application and Procedural Task Execution as source context families. At test time, each held-out context induces one generated skill, and that skill is reused across the held-out tasks under the same context. Held-out contexts from the two source families form the in-domain evaluation, while Domain Knowledge Reasoning and Empirical Discovery & Simulation form the OOD evaluation. Skill generation may access only the source context itself; held-out task answers, task rubrics, and execution traces are never exposed during skill construction. Final CL-Bench pass/fail decisions are produced by GPT-5.5 under the official task-specific rubrics.
SpreadsheetBench and tau2-bench Protocol. SpreadsheetBench is used to evaluate spreadsheet-operation skill acquisition from execution experience. We use the 400-task benchmark and construct a stratified 200/200 source/test split that preserves the original ratio between cell-level and sheet-level tasks. Source tasks are used only to collect trajectories, tool observations, and feedback for skill construction, while held-out tasks are evaluated with the full benchmark runner. tau2-bench is used to evaluate domain workflow skill reuse from experience. We use the Airline, Retail, and Telecom domains and follow the official domain split shown in Table A3. Source tasks provide the experience pool for skill construction, and held-out tasks are used only for final evaluation. We do not include banking_knowledge because it does not provide a comparably clean train/test split.
Baselines. We report four baseline families. The control baseline is No Skill. Strong prompt baselines are Anthropic Skill-Creator and Progressive Prompt Skill, both of which can operate on document/context evidence and trajectory/experience evidence through prompting alone. The document-to-skill baselines are Ctx2Skill and AutoSkill. The experience-to-skill baselines are ExpeL, AWM, Trace2Skill, SkillX, and SkillPro. All reported baselines are evaluated under the same source-evidence and held-out-task protocol as Skill-.
Fair Evaluation Protocol. For each source unit, every method receives the same source evidence and is evaluated on the same held-out tasks after injecting the generated skill into the worker agent. Document-based methods may use only source contexts, while experience-based methods may use only source trajectories, tool observations, and execution feedback. For CL-Bench, the direct-context control receives the raw context without an external generated skill. For SpreadsheetBench and tau2-bench, the source trajectories are sampled once by GPT-4o on the benchmark training split, and this trajectory source split is exactly the one used to train Skill-. All automatic experience-to-skill baselines consume this same GPT-4o trajectory pool. Retrieval-style methods are also restricted to the same source group, so the comparison does not benefit from cross-domain leakage. Main tables report GPT-4o same-worker evaluation and Claude-Sonnet-4.5 cross-worker transfer. Unless a baseline intrinsically defines another construction procedure, skill artifacts are generated once with GPT-4o and then reused for both worker backbones; Claude-Sonnet-4.5 is used only as a downstream worker and does not regenerate the skills. For CL-Bench, the final skill is appended to the original benchmark messages and the answer is graded by GPT-5.5 with the official task-specific rubrics. For SpreadsheetBench and tau2-bench, the final skill is inserted above the official benchmark task prompt while the original environment harness, action parser, and checker remain unchanged. The local anchored-query comparison described in Section 4.2 is used only to compute RL training rewards, not the reported test metrics. No held-out queries, test rubrics, test trajectories, or test feedback are used for skill construction, reward design, or model selection. All methods are therefore compared under the same benchmark-side success criteria and worker-side prompting interface.
Appendix D Prompts
This section records the concrete prompt interfaces that define the training and evaluation protocol. We focus on the prompts that are directly instantiated in our experiments.
D.1 Skill- Core Prompts
Skill- System Prompt.
This shared interface is used for SFT, GRPO rollout, and final skill generation. The user message is unchanged across document/context evidence and trajectory/experience evidence; only the serialized payload differs across benchmarks.
Teacher Prompt for SFT Data Construction.
In the finalized warm-up pipeline, this teacher role is instantiated by DeepSeek-V4-Pro. The accepted demonstrations cover both expansion and maintenance actions, and repeated late-stage Noop steps are collapsed during filtering to avoid overrepresenting conservative edits.
D.2 Reward and Evaluation
CL-Bench Rubric Judge Prompt.
This GPT-5.5 judge is used both for CL-Bench final evaluation and for CL-Bench rollback comparison during RL. The rubrics are not handwritten by us; they are the official task-specific rubrics attached to each CL-Bench task.
Experience-Benchmark Skill Injection Prompt.
SpreadsheetBench and tau2-bench both preserve their original task prompts and prepend only this reusable-skill block. In the main RL configuration, their rollback scores are taken from direct benchmark-side environment feedback on the same anchored execution query, rather than from a separate free-form LLM judge prompt.
D.3 Baselines
Anthropic Skill-Creator Wrapper Prompt.
This wrapper is used as a strong prompt-only baseline in both document-to-skill and experience-to-skill settings. It receives exactly the same source evidence as Skill- and produces a complete skill for each bounded evidence chunk; when multiple chunks are required, the resulting artifacts are merged as described in Appendix B.2.
Progressive Prompt Skill Update Prompt.
This baseline shares the same progressive evidence exposure pattern as Skill-, but it does not learn an editing policy and does not receive rollback reward. It therefore isolates the effect of training from the effect of multi-step prompting alone.
Other Baseline Prompt Families. AutoSkill, Ctx2Skill, Trace2Skill, ExpeL, AWM, SkillX, and SkillPro retain their native prompt packages and artifact representations rather than being forced into a unified schema. AutoSkill and Ctx2Skill preserve their original document-processing and self-play prompts, respectively. The experience-based methods render the shared trajectory pool as patch-oriented skill folders, lesson lists, workflow memories, hierarchical skill libraries, or option-style skill pools, respectively. Their benchmark-specific wrappers and selection constraints are documented in Appendix B.2. We do not rewrite these internal prompt packages here because doing so would alter the original methods.
Appendix E Limitations
Skill- has three main limitations. First, Appendix A provides only a local ranking result for candidate edits under the stated assumptions. A single rollback comparison is a noisy binary signal on one anchored query, and it neither estimates an expected score difference nor guarantees that repeated local updates improve the final skill across its task family. Second, the current reward interface still relies on benchmark-specific verifier design, so transferring the method to new environments requires additional work on evaluation protocols and comparison criteria. Third, the current action space and skill representation are deliberately simple and text-centric. They work well for SKILL.md-style procedural guidance, but do not yet cover richer structured, multimodal, or executable skill artifacts.