Zi-Han Wang

Zhengxi Lu

Zhiyuan Yao

Jie Wu

Zhengzhou Cai

Yueqing Sun

Ziang Ye

Linji Hao

Yujiu Yang

zethive0225@gmail.com guqi03@meituan.com

Work done during internship at Meituan.Corresponding author

Abstract

Reinforcement learning(RL) with verifiable rewards constructs trajectory-level advantage, yet often fails to credit the few pivotal decisions that drive outcomes in long-horizon multi-turn agentic RL. Some recent works introduce privileged self-distillation into credit assignment for RL, offering denser supervision, but it still remains unclear how such a local signal should express sequential credit. We therefore propose AgentOPSD, a critic-free recursive turn-level credit assignment for agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence, recursively updates a Bayesian belief state in log-odds space. This provides a principled reweighting scheme that transforms sparse outcome supervision into turn-level credit signals and identifies pivotal turns by the marginal revision between consecutive states while remaining fully compatible with standard policy optimization, requiring neither additional rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA with two Qwen model scales (3B and 7B). AgentOPSD improves over GRPO and strong self-distillation baselines, reaching $89.1\%$ success on ALFWorld with Qwen2.5-7B, and ablations attribute the gains to turn-level aggregation and history-dependent recursive belief updates. Our code is available at https://github.com/ZethWang/AgentOPSD.

1 Introduction

Agentic post-training has become an important approach to improving the ability of large language models to solve complex tasks (Guo et al., 2025; Team et al., 2025; Yang et al., 2025; Comanici et al., 2025; Team et al., 2026b). Unlike static, single-turn reasoning, agents must continuously interact with partially observable environments, at each turn acting on the current observation and receiving a new one as the environment transitions (Shen et al., 2023; Shi et al., 2025; Jimenez et al., 2023). However, many interactive environments provide verifiable rewards only after an entire trajectory terminates, forcing training algorithms to infer the contribution of each intermediate decision from a single sparse outcome. The decisions within a trajectory can nevertheless play substantially different roles: even successful trajectories may contain spurious, redundant, or misleading actions, whereas failed trajectories may still include useful reasoning.

Group-relative policy optimization methods such as GRPO (Shao et al., 2024; Yu et al., 2025) and its agentic variants (Dong et al., 2025; Feng et al., 2025) construct a trajectory-level advantage from outcome rewards and broadcast it uniformly across the trajectory. Such uniform credit cannot distinguish a few pivotal decisions from routine operations. This limitation becomes increasingly pronounced as the interaction horizon grows. Turn-level credit assignment is therefore essential for identifying the decisions that meaningfully influence the outcome and providing more precise supervision throughout long-horizon interactions.

A complementary line of work provides denser, token-level supervision. On-policy distillation (Ye et al., 2026a; Yang et al., 2026b; Team et al., 2026a) trains a student on its own rollouts under a teacher, while its self-distillation variants (Zhao et al., 2026; He et al., 2026) remove the need for a separate teacher by conditioning the same policy on privileged information available only during training (Lu et al., 2026c). Recent studies further incorporate OPSD signals into reinforcement learning as an auxiliary source of supervision (Lu et al., 2026b; Wang et al., 2026a).

However, applying OPSD to agentic reinforcement learning introduces two mismatches. First, OPSD’s token-level signals are not naturally aligned with agentic interaction (Lu et al., 2026b), where multiple tokens jointly form an action and the environment responds only at turn boundaries. Second, even existing step-aware methods consider each turn in isolation (Zhang et al., 2026), without accounting for the evidence accumulated through preceding interactions. The central challenge is therefore to transform local OPSD signals into history-dependent turn-level credit.

Our key insight is that the credit of a turn should be determined not by its local signal in isolation, but by how much that signal changes the estimated probability of eventual success. To formalize this intuition, we interpret the per-turn self-distillation gap as new evidence that induces a Bayesian belief update (Åström, 1965; Kaelbling et al., 1998). We define the corresponding belief state as the probability that the trajectory will ultimately succeed given the interaction history.

Based on this insight, we propose AgentOPSD (Recursive Self-Distillation for Agentic Reinforcement Learning), a turn-level credit-assignment method for long-horizon agents. AgentOPSD aggregates token-level teacher–student log-probability gaps into turn-level evidence. Starting from the average group success rate, it then recursively updates Bayesian belief state at each turn in log-odds space without additional rollouts or a learned critic. The outcome verifier determines the global direction of optimization, while the bayesian belief updates redistribute the trajectory-level learning signal across turns. We evaluate AgentOPSD on three interactive environments—ALFWorld (Shridhar et al., 2020), WebShop (Yao et al., 2022), and Search-QA (Jin et al., 2025)—and across two model scales. As shown in Figure 1, AgentOPSD consistently outperforms GRPO and strong self-distillation baselines. Further ablations show that gains from aggregating token-level signals at turn boundaries aligned with environment transitions, and transforming an isolated local gap into a recursive revision of belief.

• We formalize turn-level credit as the revision of a success belief induced by each turn. In log-odds space, this connects per-turn evidence to a recursive Bayesian update and reveals that an isolated self-distillation gap is not, by itself, sequential credit.

• We introduce AgentOPSD, which aggregates token-level teacher–student log-probability gaps into environment-aligned turn-level evidence and recursively propagates this evidence through the trajectory-success belief, without additional rollouts or a learned critic.

• Experiments and ablations across three interactive environments and two model scales demonstrate that AgentOPSD consistently outperforms GRPO and strong self-distillation baselines. Ablations further verify the complementary benefits of turn-boundary aggregation and recursive belief revision.

2 Methodology

2.1 Problem Setup

Given a task $x$ and initial observation $o_0$, the agent starts from $s_1=(x,o_0)$. At turn $k$, it samples where ${\pi }_{\theta }$ is current policy, $s_k$ is its visible interaction history, $y_{k,t}$ is the $t$-th token of action $a_k$, and $L_k$ is the action length. After observing $o_k$, the history becomes $s_{k+1}=(s_k,a_k,o_k)$. A $K$-turn episode forms $𝝉=(s_1,a_1,o_1,\dots ,s_K,a_K,o_K)$ and receives a binary outcome reward $R\left(𝝉\right)$.

$a_k=(y_{k,1},\dots ,y_{k,L_k})\sim {\pi }_{\theta }(\cdot \mid s_k),$

(1)

For each task, group-relative policy optimization samples $G$ trajectories and computes the sequence-level advantage. Here $i$ indexes one of the $G$ sampled trajectories, while $\overline{R}$ and ${\hat{\sigma }}_R$ are the group reward mean and standard deviation, and ${ϵ}_0$ is a small positive constant (reused throughout to avoid division by zero or infinite log-odds). GRPO assigns $A_{\mathrm{seq}}^{\left(i\right)}$ to every token in trajectory $i$, leaving turn-level credit unresolved.

$A_{\mathrm{seq}}^{\left(i\right)}=\frac{R^{\left(i\right)}-\overline{R}}{{\hat{\sigma }}_R+{ϵ}_0},\overline{R}=\frac{1}{G}\sum _{j=1}^G{R}^{\left(j\right)}.$

(2)

2.2 From Outcome Contribution to Bayesian Turn Evidence

Directly measuring the counterfactual contribution of turn $k$ would require marginalizing the outcome reward over all possible continuations following $a_k$, which is intractable in long-horizon interactions. We therefore adopt a hindsight-based evidential perspective. Let $C$ denote the event that the trajectory eventually succeeds. If $a_k$ supports success, it should be more characteristic of successful behavior than of unsuccessful behavior. Bayes’ rule expresses the resulting change in the belief about $C$ as an action-side likelihood ratio (Åström, 1965; Kaelbling et al., 1998):

$\mathrm{logit}p\left(C\mid s_k,a_k\right)-\mathrm{logit}p\left(C\mid s_k\right)=\log \frac{p\left(a_k\mid s_k,C\right)}{p\left(a_k\mid s_k,\neg C\right)}.$

(3)

Here $\mathrm{logit}\left(u\right)=\log \frac{u}{1-u}$. The right-hand side is the ideal Bayes factor (Kass and Raftery, 1995) between the success-conditional and failure-conditional likelihoods of $a_k$. Its sign indicates whether the action increases or decreases support for eventual success.

Because these outcome-conditional behavioral distributions are not directly available, we estimate this Bayesian evidence retrospectively using a computable per-turn self-distillation contrast between a privileged, success-associated self-teacher and the standard student policy. This contrast provides a tractable proxy for the otherwise inaccessible belief update; we characterize the approximation and its conditions below. The student and teacher share parameters $\theta$ and score the same student-generated action (Zhao et al., 2026). Their token contexts are

$h_{k,t}=(s_k,y_{k,<t}),h_{k,t}^{+}=(s_k,c^{+},y_{k,<t}).$

(4)

where $y_{k,<t}$ is the token prefix within turn $k$, and $c^{+}$ is a training-only retrieved skill describing useful subgoals and action patterns (Xia et al., 2026). The skill-conditioned branch approximates success-associated behavior, while the unconditioned branch provides the background likelihood.

For token $y_{k,t}$, define the detached likelihood contrast

${\delta }_{k,t}=\log {\pi }_{\theta }\left(y_{k,t}\mid h_{k,t}^{+}\right)-\log {\pi }_{\theta }\left(y_{k,t}\mid h_{k,t}\right).$

(5)

Positive ${\delta }_{k,t}$ means that $c^{+}$ increases the likelihood of the generated token. Summing over the $L_k$ tokens gives the turn-level evidence

$e_k=\sum _{t=1}^{L_k}{\delta }_{k,t}=\log \frac{{\pi }_{\theta }\left(a_k\mid s_k,c^{+}\right)}{{\pi }_{\theta }\left(a_k\mid s_k\right)}.$

(6)

Accordingly, $e_k$ provides a tractable hindsight approximation to the ideal Bayesian turn evidence in Eq. (3), under the conditions detailed in Appendix A.1. More generally, Bayes’ rule gives

$\log \frac{p\left(a_k\mid s_k,C\right)}{p\left(a_k\mid s_k\right)}=\log \frac{p\left(C\mid s_k,a_k\right)}{p\left(C\mid s_k\right)}.$

(7)

Thus, $e_k$ can be interpreted as an evidential score whose sign indicates whether $a_k$ raises or lowers support for eventual success. This sign-consistent Bayesian evidence is precisely the property that AgentOPSD relies on. We therefore treat $e_k$ as a tractable Bayesian-inspired evidence proxy.

2.3 Recursive Belief update

The local score $e_k$ does not indicate whether the same evidence is pivotal or redundant given earlier turns. We therefore maintain a decaying evidence accumulator and measure each turn by how much it revises the current support state:

	$B_0$	$=\mathrm{clip}(\overline{R},{ϵ}_0,1-{ϵ}_0),$	$c_0$	$=0,$		(8)
	$c_k$	$=\gamma c_{k-1}+e_k,$	${\ell }_k$	$=\mathrm{logit}\left(B_0\right)+c_k=\mathrm{logit}\left(B_0\right)+\sum _{j=1}^k{\gamma }^{k-j}{e}_j,$

with $B_k=\sigma \left({\ell }_k\right)$ and $\sigma \left(u\right)={\left(1+e^{-u}\right)}^{-1}$. Here $\overline{R}=S/G$ is the fraction of successful trajectories in the group of size $G$—the standard GRPO group mean (Prop. 7)—and $B_0$ clips it to $[{ϵ}_0,1-{ϵ}_0]$ with ${ϵ}_0={10}^{-4}$ so its log-odds stay finite for all-correct or all-wrong groups. $c_k$ is the accumulated evidence, and $\gamma \in (0,1]$ is a decay factor that down-weights older turns geometrically; only the evidence $c_k$ decays, while the prior $\mathrm{logit}\left(B_0\right)$ is retained at every step. Setting $\gamma =1$ recovers the undiscounted accumulation of a log-likelihood ratio familiar from sequential testing (Wald, 1945); $\gamma <1$ makes the state recency-weighted, so that evidence from many turns ago no longer pins the support level. Since $e_k$ is estimated by the self-teacher (§2.2), $B_k$ is treated as relative support rather than a calibrated success probability.

The importance of turn $k$ is its marginal support revision:

	$\Delta B_k$	$=B_k-B_{k-1}=\sigma \left({\ell }_k\right)-\sigma \left({\ell }_{k-1}\right),$		(9)
	$\Delta B_k$	$\approx B_{k-1}\left(1-B_{k-1}\right)\left(e_k-\left(1-\gamma \right)c_{k-1}\right).$

The increment ${\ell }_k-{\ell }_{k-1}=e_k-\left(1-\gamma \right)c_{k-1}$ is the new evidence net of the decayed carry-over, and it is weighted by the current state sensitivity $B_{k-1}\left(1-B_{k-1}\right)$: evidence has greatest effect under uncertainty and is suppressed once support saturates. At $\gamma =1$ this reduces to $B_{k-1}\left(1-B_{k-1}\right)e_k$. We update at turn boundaries; a token-level variant is used only as an ablation.

Outcome-aligned recursive credit.

We align the revision with the terminal update and read off its magnitude and direction:

$q_k=\mathrm{sign}\left(A_{\mathrm{seq}}\right)\Delta B_k.$

(10)

The magnitude $\left|\Delta B_k\right|=\left|q_k\right|$ measures how much support the turn revises, while its sign $\mathrm{sign}\left(q_k\right)$ records whether that revision agrees with the verifier’s outcome signal. After the within-trajectory standardization below, turns with above-average $q_k$ are amplified and those below-average are attenuated; since the multiplier stays strictly positive, this never reverses the GRPO update direction.

2.4 Bounded Advantage Reshaping

The raw credit $q_k$ only modulates the magnitude of the verifier-derived advantage. For trajectory $i$, we normalize its $K_i$ turn credits and apply a bounded multiplier:

	${\mu }_q^{\left(i\right)}$	$=K_i^{-1}\sum _{j=1}^{K_i}{q}_j^{\left(i\right)},$	${\sigma }_q^{\left(i\right)}$	$=\sqrt{K_i^{-1}\sum _{j=1}^{K_i}{\left(q_j^{\left(i\right)}-{\mu }_q^{\left(i\right)}\right)}^2},$		(11)
	$z_k^{\left(i\right)}$	$=\frac{q_k^{\left(i\right)}-{\mu }_q^{\left(i\right)}}{{\sigma }_q^{\left(i\right)}+{ϵ}_0},$	$w_k^{\left(i\right)}$	$=\mathrm{clip}(1+b{z}_k^{\left(i\right)},\mathrm{\hspace{0.17em}1}-b,\mathrm{\hspace{0.17em}1}+b),$
	${\tilde{A}}_k^{\left(i\right)}$	$=A_{\mathrm{seq}}^{\left(i\right)}\left[\left(1-\lambda \right)+\lambda w_k^{\left(i\right)}\right],$	$b$	$\in (0,1),\lambda \in [0,1].$

Here ${\mu }_q^{\left(i\right)}$ and ${\sigma }_q^{\left(i\right)}$ are the within-trajectory mean and standard deviation, $z_k^{\left(i\right)}$ is the normalized credit (${ϵ}_0$ stabilizes the normalization), $b$ sets $w_k^{\left(i\right)}\in [1-b,1+b]$, and $\lambda$ controls reshaping strength.

Token $t$ inherits ${\tilde{A}}_{{\kappa }_i\left(t\right)}^{\left(i\right)}$, yielding

	${ℒ}_{\text{AgentOPSD}}\left(\theta \right)$	$=-\frac{1}{G}\sum _{i=1}^G\frac{1}{{\sum }_t{M}_{i,t}}\sum _t{M}_{i,t}\min (r_{i,t}{\tilde{A}}_{{\kappa }_i\left(t\right)}^{\left(i\right)},\mathrm{clip}(r_{i,t},1-\epsilon ,1+\epsilon ){\tilde{A}}_{{\kappa }_i\left(t\right)}^{\left(i\right)})+\beta {ℒ}_{\mathrm{KL}},$		(12)
	$r_{i,t}$	$=\frac{{\pi }_{\theta }\left(y_{i,t}\mid h_{i,t}\right)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(y_{i,t}\mid h_{i,t}\right)}.$

Here $M_{i,t}\in \{0,1\}$ masks valid response tokens, ${\kappa }_i\left(t\right)$ maps token $t$ to its turn, $r_{i,t}$ is the importance ratio against the rollout policy ${\pi }_{{\theta }_{\mathrm{old}}}$, $\epsilon$ is the clipping radius, and $\beta$ is its coefficient. No separate distillation loss is introduced; the detached self-teacher signal acts only through $\tilde{A}$.

媒体内容 · 前往原文查看

	ALFWorld							Search-QA								WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg	NQ	Triv	Pop	Hotp	2Wk	MuS	Bam	Avg	Score	Acc
Qwen2.5-3B-Instruct
Vanilla	44.4	11.1	6.2	15.4	28.6	12.5	21.9	24.6	48.1	31.0	26.3	25.3	7.2	59.7	31.7	6.7	0.8
Skill-Prompt*	51.7	66.7	48.4	0.0	4.3	10.0	28.9	23.7	46.2	30.6	24.4	22.1	7.5	12.5	23.9	0.2	0.8
OPSD	48.8	41.7	16.7	0.0	15.8	16.7	28.1	0.1	0.1	0.1	0.0	0.0	0.0	0.0	0.0	11.3	3.1
GRPO	91.2	62.5	96.2	61.9	65.0	47.4	75.0	39.3	60.6	41.1	37.4	34.6	15.4	26.4	36.4	79.8	63.3
Skill-GRPO	88.9	71.4	58.8	70.6	40.7	29.2	60.2	43.5	58.8	43.0	36.8	32.2	11.7	12.5	34.1	77.3	60.9
Skill-GRPO*	94.3	57.1	100	66.7	73.1	57.1	80.5	44.3	59.6	44.3	39.0	36.1	14.5	14.9	36.1	76.3	66.4
GRPO+OPSD	100	82.4	85.7	75.0	70.0	60.0	81.2	44.9	61.2	45.2	40.4	38.5	16.0	66.1	44.6	77.8	66.4
Skill-SD	88.2	50.0	96.2	52.4	65.0	57.9	73.4	44.4	60.4	44.0	39.5	40.4	15.4	64.9	44.1	75.9	64.0
RLSD	87.9	75.0	90.9	75.0	73.1	68.4	79.7	41.5	58.6	42.3	40.4	40.2	16.8	66.9	43.8	84.4	66.4
SDAR	97.1	62.5	100	61.9	75.0	84.2	84.4	44.8	58.1	44.3	38.6	36.2	15.7	66.1	43.4	85.0	68.0
StepOPSD	82.4	66.7	82.6	52.2	73.7	75.0	73.4	43.6	61.2	43.8	39.2	38.1	15.8	64.5	43.7	82.4	66.4
AgentOPSD	100	68.8	86.4	66.7	84.6	73.7	84.4	44.9	61.9	48.0	40.9	41.6	14.4	68.1	46.7	90.4	69.5
Qwen2.5-7B-Instruct
Vanilla	36.1	22.2	3.1	0.0	0.0	0.0	12.5	25.2	50.8	29.5	29.0	29.0	10.4	63.7	33.9	5.9	1.6
Skill-Prompt*	51.7	50.0	32.3	5.3	4.3	0.0	23.4	30.9	52.1	32.7	32.7	27.9	12.7	66.1	36.4	1.7	0.8
OPSD	50.0	60.0	22.7	21.4	17.6	9.5	32.8	8.8	8.6	17.5	2.5	4.2	0.5	1.2	6.2	4.5	2.3
GRPO	91.2	87.5	96.2	81.0	65.0	57.9	81.2	45.1	63.7	44.0	43.6	43.2	16.8	37.6	42.0	80.9	72.6
Skill-GRPO	88.5	66.7	65.2	61.1	57.7	73.1	69.5	45.2	63.7	45.7	43.1	43.3	19.6	21.4	40.3	80.4	71.9
Skill-GRPO*	100	83.3	96.4	83.3	75.0	78.9	88.3	44.8	63.0	45.1	43.7	43.7	20.5	71.4	47.5	87.0	81.2
GRPO+OPSD	91.4	61.5	100	87.5	76.5	52.2	80.4	47.3	64.5	46.9	43.8	39.3	18.0	69.4	47.0	86.8	76.5
Skill-SD	93.9	93.8	90.9	100	69.2	68.4	85.1	47.1	64.5	47.8	44.2	42.1	20.2	69.0	47.8	86.1	76.5
RLSD	100	87.5	92.3	58.8	80.0	65.2	82.0	46.8	63.0	44.4	45.5	48.9	21.5	73.0	49.0	87.4	77.3
SDAR	94.7	75.0	100	86.7	68.2	78.9	85.9	46.3	63.5	48.2	43.8	48.4	19.6	73.0	49.0	89.4	82.8
StepOPSD	98.1	75.0	100	90.5	80.0	63.2	88.4	45.3	64.6	45.1	44.5	44.4	19.3	69.8	48.2	87.2	78.1
AgentOPSD	91.2	87.5	100	90.5	75.0	84.2	89.1	47.5	64.9	46.8	45.8	45.3	20.9	70.2	49.2	90.2	79.7

3 Experiments

3.1 Experimental Setup

Benchmarks.

We evaluate on three environments. ALFWorld (Shridhar et al., 2020) is a text embodied benchmark over six household task categories—Pick and Place (Pick), Look at Object in Light (Look), Pick Clean then Place (Clean), Pick Heat then Place (Heat), Pick Cool then Place (Cool), and Pick Two and Place (Pick2). Search-QA follows the Search-R1 setup (Jin et al., 2025) and covers single-hop QA (NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023)) and multi-hop QA (HotpotQA (Yang et al., 2018), 2Wiki (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), Bamboogle (Press et al., 2023)), with NQ and HotpotQA in-domain and the rest held out; retrieval uses E5 (Wang et al., 2022). WebShop (Yao et al., 2022) is an interactive online-shopping environment; we evaluate on the 128 fixed validation tasks of Feng et al. (2025).

Implementation.

We train Qwen2.5-3B/7B-Instruct on $8\times$H800 GPUs. The privileged skills are retrieved from the SkillBank of SkillRL (Xia et al., 2026) by keyword matching and are used only during training; inference uses no external skills. Prior $B_0$ set to the fraction of successful trajectories in each GRPO group (the standard group mean $\overline{R}$). All other optimization settings are shared with the SDAR baseline. Full training and AgentOPSD hyperparameters are listed in Appendix F (Table 3).

Baselines.

We compare against three groups. (1) Training-free: Vanilla (the base model) and Skill-Prompt, which prepends retrieved skills at inference. (2) Group-relative RL: GRPO (Shao et al., 2024) and Skill-GRPO, which injects skills into the training prompt (evaluated with, Skill-GRPO*, or without retrieved skills). (3) Self-distillation RL: OPSD (Zhao et al., 2026), GRPO+OPSD, Skill-SD (Wang et al., 2026a), RLSD (Yang et al., 2026a), SDAR (Lu et al., 2026b) and StepOPSD (Zhang et al., 2026), all of which use the teacher–student gap but inject it as a gate, magnitude, or auxiliary loss. All methods share the same backbone, data, and budget. Full algorithm details are in Appendix B.

3.2 Main Results

The gain comes from credit construction, not privileged access.

Under our unified setup, AgentOPSD and the privileged baselines use the same retrieved skills; they differ primarily in how the skill-induced teacher–student discrepancy enters learning. AgentOPSD outperforms GRPO+OPSD, Skill-SD, and RLSD on all eight aggregate comparisons across the two model scales, and exceeds SDAR on six of eight. This controlled-information comparison isolates the benefit of AgentOPSD: a local teacher-student gap is not yet a reliable credit signal. Accumulating that gap into a belief state and assigning credit according to belief revision more effectively identifies the turns that change the predicted outcome.

The advantage grows with the interaction horizon.

AgentOPSD is designed for the regime where uniform credit is most harmful, so we also ask how performance degrades as tasks require more turns. Figure 1(b) regresses per-sub-task success on the measured mean number of turns of successful episodes on ALFWorld (Qwen2.5-7B), reporting the success points lost per additional turn. The uniform-credit methods degrade fastest ($-3.59$ for RLSD and $-2.91$ for GRPO points per turn), whereas AgentOPSD is the flattest at $-0.54$. This is consistent with the motivation for turn-level credit: the longer the trajectory, the more decisions a single broadcast advantage has to cover, and the more a history-dependent revision helps.

3.3 Mechanism Ablation

Table 2 evaluates each design choice on ALFWorld with Qwen2.5-7B by removing or replacing one component at a time. The full method achieves a success rate of $89.1$.

媒体内容 · 前往原文查看

Component	Ablation	ALFWorld
AgentOPSD (full)	turn-level, bounded, $\lambda =0.5$	89.1
Turn-level granularity	per-token accumulation	85.9
Recursive state revision (8)	raw local gap $e_k$ in place of $\Delta B_k$	82.8
Signed direction (10)	magnitude $\left|\Delta B_k\right|$ only (drop outcome sign)	80.5
State prior $B_0$ anchor	drop empirical-rate initialization	78.9

Granularity and recursion.

Replacing turn-level belief tracking with per-token accumulation reduces the success rate to $85.9$: environment feedback is associated with a complete action rather than an individual token, so token-level accumulation fragments a single decision and weakens the alignment between the gap and outcomes. Replacing the recursive revision $\Delta B_k$ with the raw local gap $e_k$ further reduces performance to $82.8$. A raw $e_k$ scores each turn in isolation, whereas $\Delta B_k$ measures how that gap revises the belief state accumulated over the preceding history—so the same local gap that is decisive while the outcome is open becomes redundant once the accumulated state already points to an outcome. This controlled comparison isolates the value of the recursion and directly confirms our central principle that a local gap is not sequential credit.

Outcome-aligned signed direction.

Keeping only the magnitude $\left|\Delta B_k\right|$ and dropping the sign (Eq. 10), i.e. standardizing $\left|\Delta B_k\right|$ instead of the signed $q_k$, lowers performance to $80.5$. The magnitude identifies where the belief state changes, but cannot determine whether that change agrees with the verifier outcome. For a successful trajectory, an upward belief revision is consistent with the outcome, whereas for a failed trajectory the same revision is inconsistent. The signed direction makes this distinction explicit, allowing outcome-consistent revisions to receive more credit and contradictory revisions to receive less.

State-prior anchoring.

Removing the empirical prior $B_0=\mathrm{clip}(\overline{R},{ϵ}_0,1-{ϵ}_0)$ reduces the success rate to $78.9$. The group success rate $\overline{R}$ provides a verifier-grounded estimate of task difficulty before the trajectory-specific gap is accumulated. Moreover, $B_0$ determines the initial log-odds and thus the operating region of the $B\left(1-B\right)$ gate. Without this anchor, trajectories begin from an arbitrary uncertainty level, which can mis-scale early belief revisions and distort which early turns appear pivotal. The ablations therefore separate three roles: belief revision localizes credit, the signed direction aligns it with the final outcome, and prior anchoring stabilizes its reference point.

3.4 Hyperparameter Sensitivity

Whereas the mechanism ablation asks whether each component is necessary, we now examine how sensitive AgentOPSD is to its continuous hyperparameters. We sweep one knob at a time while holding the others at the full-AgentOPSD setting ($\lambda =0.5$, $\gamma =0.95$, ${ϵ}_{\mathrm{high}}=0.24$; Appendix F) across three configurations (Figure 3): long-horizon ALFWorld with Qwen2.5-7B ($89.1$) and Qwen2.5-3B ($84.4$), and short-horizon Search-QA with Qwen2.5-3B ($46.7$).

Reshaping weight $\lambda$.

Sweeping $\lambda \in \{0.5,0.25,0.1,0.01\}$ interpolates between pure GRPO ($\lambda =0$) and full belief reshaping. This is the knob with the clearest effect: $\lambda =0.5$ is best and any smaller value reduces performance ($89.1$ at $\lambda =0.5$ vs. $84.4/85.9/83.6$; Search $46.7$ vs. $45.1/40.2/45.4$), consistent with a smaller $\lambda$ down-weighting the bounded multiplier and discarding turn-level credit. We use $\lambda =0.5$ throughout.

Evidence decay $\gamma$.

Turn-level evidence is accumulated with a geometric decay $c_k=\gamma c_{k-1}+e_k$ (equivalently ${\ell }_k=\mathrm{logit}\left(B_0\right)+{\sum }_{j\le k}{\gamma }^{k-j}{e}_j$); sweeping $\gamma \in \{1.0,0.95,0.9,0.8\}$ moves the result within a few points ($87.5/82.0/85.2$; Search $45.1/44.5/45.5$) without a monotone trend, so the recursion is not particularly sensitive to how fast old evidence is discounted. We use the mild setting $\gamma =0.95$ for all main results.

Policy clipping ${ϵ}_{\mathrm{high}}$.

Fixing ${ϵ}_{\mathrm{low}}=0.2$ and varying ${ϵ}_{\mathrm{high}}\in \{0.2,0.24,0.28\}$ (clip-higher (Yu et al., 2025)), AgentOPSD is largely unaffected ($88.3$ at both $0.2$ and $0.28$; Search $46.9$ and $45.7$), indicating that the reshaped objective inherits the trust-region robustness of GRPO. Overall, only $\lambda$ produces a systematic effect, and the spread across all knobs shrinks sharply on the four-turn Search-QA task—the settings that matter on long-horizon ALFWorld are largely inert when little history accumulates, which is again consistent with the method acting where long-horizon credit assignment is needed.

4 Related Work

4.1 Agentic Post-Training with Verifiable Rewards

Reinforcement learning with verifiable rewards has advanced from single-turn reasoning (Shao et al., 2024; Guo et al., 2025; Yu et al., 2025; Lu et al., 2026a) to long-horizon agents in embodied text worlds, web shopping, and retrieval-augmented question answering (Shridhar et al., 2020; Yao et al., 2022; Jin et al., 2025; Lu et al., 2025). In these interactive settings, sparse terminal rewards make turn-level credit assignment particularly challenging. Standard GRPO broadcasts a trajectory-level advantage uniformly across turns. GiGPO (Feng et al., 2025) improves reward-side credit assignment by combining episode-level advantages with step-level advantages estimated from repeated anchor states across trajectories. In contrast, AgentOPSD derives turn-level evidence from privileged teacher–student likelihood gaps and assigns credit through recursive belief revision. GiGPO and AgentOPSD therefore operate on complementary signal sources—environment rewards and self-distillation evidence, respectively.

4.2 On-Policy (Self-)Distillation

On-policy distillation trains a policy on its own rollouts under a teacher (Agarwal et al., 2024; Gu et al., 2026; Wen et al., 2023). Its recent self-distillation variants remove the need for a separate teacher while the teacher branch is conditioned on privileged information available only during training (Zhao et al., 2026; He et al., 2026; Lu et al., 2026c). Recent studies incorporate the resulting teacher–student log-probability gap into RLVR by using it to scale or reshape the advantage (Yang et al., 2026a), as a detached auxiliary objective (Lu et al., 2026b; Wang et al., 2026a), or as reweighted, scheduled, or reward-densifying local supervision (Xu et al., 2026; Wang et al., 2026b; Ye et al., 2026a; He et al., 2026). Existing methods predominantly treat the distillation gap as a local token-level or step-level signal. Token-level signals are not naturally aligned with action turns, and the contribution of a turn depends on the evidence accumulated through preceding interactions. StepOPSD (Zhang et al., 2026) aggregates the teacher–student signal over action-centered step spans but still scores each span by its local log-ratio. In contrast, AgentOPSD first aggregates token-level gaps within each turn and then recursively accumulates the resulting evidence into a running support state.

4.3 Long-Horizon Credit Assignment

Assigning credit across a long horizon is a classical problem. PPO learns a value function and, via GAE, derives a per-step temporal-difference signals (Schulman et al., 2017; 2016). When rewards are sparse and delayed, return-decomposition methods such as RUDDER redistribute a terminal reward to the steps responsible for it (Arjona-Medina et al., 2019), while process reward models and Monte-Carlo credit methods such as VinePPO estimate intermediate value by additional rollouts or a learned scorer (Cui et al., 2025; Kazemnejad et al., 2024). These approaches recover per-step structure but reintroduce the cost GRPO removed: a trained critic, a reward model, or many extra rollouts. AgentOPSD restores a per-turn value signal in the critic-free group-relative setting, at the cost of a single teacher forward pass. The belief state plays the role of GAE’s value baseline and its per-turn revision the role of the TD signal, but without a learned value network cost.

5 Conclusion

We studied credit assignment for long-horizon language agents, where trajectory-level rewards provide limited supervision for distinguishing pivotal decisions from routine or redundant actions. Our key insight is that turn-level credit should depend not only on a local signal, but also on how that signal revises the accumulated belief in eventual trajectory success. Based on this insight, we proposed AgentOPSD, which aggregates token-level self-distillation gaps at environment-aligned turn boundaries and recursively updates a trajectory-success belief in log-odds space. These belief revisions redistribute the trajectory-level advantage across turns without requiring additional rollouts or a learned critic. Experiments across three interactive environments and two model scales show that AgentOPSD consistently outperforms GRPO and strong self-distillation baselines. Ablations further confirm the importance of both turn-level signal aggregation and history-dependent belief revision. Overall, our results suggest that recursive belief updating provides a simple and effective approach to restoring fine-grained temporal credit in critic-free agentic reinforcement learning.

References

• R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. External Links: 2306.13649, Link Cited by: §4.2.
• J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter (2019) RUDDER: return decomposition for delayed rewards. External Links: 1806.07857, Link Cited by: §4.3.
• K. J. Åström (1965) Optimal control of markov processes with incomplete state information i. Journal of Mathematical Analysis and Applications 10, pp. 174–205. External Links: Document Cited by: §1, §2.2.
• Y. Chen, Z. Cai, X. Ji, W. Zhao, A. Zhang, X. Wang, and T. Chua (2026a) Understanding multilingualism in mixture-of-experts llms: routing mechanism, expert specialization, and layerwise steering. arXiv preprint arXiv:2601.14050. Cited by: Appendix C.
• Y. Chen, Y. Wang, Y. Zhang, Z. Ye, Z. Cai, Y. Shi, Q. Gu, H. Su, X. Cai, X. Wang, et al. (2026b) Learning to self-verify makes language models better reasoners. arXiv preprint arXiv:2602.07594. Cited by: Appendix C.
• G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1.
• G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, J. Yuan, H. Chen, K. Zhang, X. Lv, S. Wang, Y. Yao, X. Han, H. Peng, Y. Cheng, Z. Liu, M. Sun, B. Zhou, and N. Ding (2025) Process reinforcement through implicit rewards. External Links: 2502.01456, Link Cited by: §4.3.
• G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, et al. (2025) Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. Cited by: §1.
• L. Feng, Z. Xue, T. Liu, and B. An (2025) Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978. Cited by: Appendix C, §1, §3.1, §4.1.
• Y. Gu, L. Dong, F. Wei, and M. Huang (2026) MiniLLM: on-policy distillation of large language models. External Links: 2306.08543, Link Cited by: §4.2.
• D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §4.1.
• Y. He, S. Kaur, A. Bhaskar, Y. Yang, J. Liu, N. Ri, L. Fowl, A. Panigrahi, D. Chen, and S. Arora (2026) Self-distillation zero: self-revision turns binary rewards into dense supervision. External Links: 2604.12002, Link Cited by: §1, §4.2.
• X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609–6625. Cited by: Appendix C, §3.1.
• X. Ji, Y. Chen, Z. Cai, X. Wang, A. Zhang, and T. Chua (2026) Tiny brains, giant impact: uncovering the keystone neurons of llm with just a few prompts. arXiv preprint arXiv:2605.24846. Cited by: Appendix C.
• C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023) Swe-bench: can language models resolve real-world github issues?. arXiv preprint arXiv:2310.06770. Cited by: §1.
• B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Cited by: Appendix C, §1, §3.1, §4.1.
• M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017) Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611. Cited by: Appendix C, §3.1.
• L. P. Kaelbling, M. L. Littman, and A. R. Cassandra (1998) Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1–2), pp. 99–134. External Links: Document Cited by: §1, §2.2.
• R. E. Kass and A. E. Raftery (1995) Bayes factors. Journal of the American Statistical Association 90 (430), pp. 773–795. External Links: Document Cited by: §2.2.
• A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. L. Roux (2024) VinePPO: refining credit assignment in rl training of llms. External Links: 2410.01679, Link Cited by: §4.3.
• T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. (2019) Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, pp. 453–466. Cited by: Appendix C, §3.1.
• Z. Lu, Y. Chai, Y. Guo, X. Yin, L. Liu, H. Wang, H. Xiao, S. Ren, P. Zhao, G. Liu, et al. (2026a) Ui-r1: enhancing efficient action prediction of gui agents by reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, pp. 17608–17616. Cited by: §4.1.
• Z. Lu, Z. Yao, Z. Han, Z. Wang, J. Wu, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, et al. (2026b) Self-distilled agentic reinforcement learning. arXiv preprint arXiv:2605.15155. Cited by: Appendix D, §1, §1, §3.1, §4.2.
• Z. Lu, Z. Yao, J. Wu, C. Han, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen (2026c) SKILL0: in-context agentic reinforcement learning for skill internalization. External Links: 2604.02268, Link Cited by: §1, §4.2.
• Z. Lu, J. Ye, F. Tang, Y. Shen, H. Xu, Z. Zheng, W. Lu, M. Yan, F. Huang, J. Xiao, et al. (2025) Ui-s1: advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11543. Cited by: §4.1.
• A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2023) When not to trust language models: investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 9802–9822. Cited by: Appendix C, §3.1.
• O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis (2023) Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687–5711. Cited by: Appendix C, §3.1.
• J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2016) High-dimensional continuous control using generalized advantage estimation. External Links: 1506.02438, Link Cited by: §4.3.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §4.3.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix D, §1, §3.1, §4.1.
• Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang (2023) Hugginggpt: solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems 36, pp. 38154–38180. Cited by: §1.
• Z. Shi, S. Gao, L. Yan, Y. Feng, X. Chen, Z. Chen, D. Yin, S. Verberne, and Z. Ren (2025) Tool learning in the wild: empowering language models as automatic tool agents. In Proceedings of the ACM on Web Conference 2025, pp. 2222–2237. Cited by: §1.
• M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2020) Alfworld: aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768. Cited by: Appendix C, §1, §3.1, §4.1.
• C. Team, B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, G. Xie, H. Zhang, H. Lv, H. Li, H. Chen, H. Xu, H. Zhang, H. Liu, J. Duo, J. Wei, J. Xiao, J. Dong, J. Shi, J. Hu, K. Bao, K. Zhou, L. Li, L. Zhao, L. Zhang, P. Li, Q. Chen, S. Liu, S. Yu, S. Cao, S. Chen, S. Yu, S. Liu, T. Zhou, W. Su, W. Wang, W. Ma, X. Deng, B. Mao, B. Ye, C. Cai, C. Wang, C. Zhu, C. Ma, C. Chen, C. Li, D. Zhu, D. Xiao, D. Zhang, D. Zhang, F. Liu, F. Yang, F. Shi, G. Wang, H. Tian, H. Wu, H. Qu, H. Yi, H. An, H. Guan, X. Zhang, Y. Song, Y. Yan, Y. Zhao, Y. Lai, Y. Gao, Y. Cheng, Y. Tian, Y. Wang, Z. Tang, Z. Tang, Z. Wen, Z. Song, Z. Zheng, Z. Jiang, J. Wen, J. Sun, J. Li, J. Xue, J. Xia, K. Fang, M. Zhu, N. Chen, Q. Tu, Q. Zhang, Q. Wang, R. Li, R. Ma, S. Zhang, S. Wang, S. Li, S. Gu, S. Ren, S. Deng, T. Guo, T. Lu, W. Zhuang, W. Zhang, W. Xiong, W. Huang, W. Yang, X. Zhang, X. Yong, X. Wang, X. Xie, Y. Jiang, Y. Yang, Y. He, Y. Tu, Y. Dong, Y. Liu, Y. Ma, Y. Yu, Y. Xiang, Z. Huang, Z. Lin, Z. Xu, Z. Chen, Z. Deng, Z. Zhang, and Z. Yue (2026a) MiMo-v2-flash technical report. External Links: 2601.02780, Link Cited by: §1.
• K. Team, Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, et al. (2025) Kimi k2: open agentic intelligence. arXiv preprint arXiv:2507.20534. Cited by: §1.
• M. L. Team, A. Gui, B. Li, B. Tao, B. Zhou, B. Chen, C. Zhang, C. Gao, C. Zhang, C. Han, et al. (2026b) Longcat-flash-thinking-2601 technical report. arXiv preprint arXiv:2601.16725. Cited by: §1.
• H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2022) MuSiQue: multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics 10, pp. 539–554. Cited by: Appendix C, §3.1.
• A. Wald (1945) Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics 16 (2), pp. 117–186. External Links: Document Cited by: §2.3.
• H. Wang, G. Wang, H. Xiao, Y. Zhou, Y. Pan, J. Wang, K. Xu, Y. Wen, X. Ruan, X. Chen, and H. Qi (2026a) Skill-sd: skill-conditioned self-distillation for multi-turn llm agents. External Links: 2604.10674, Link Cited by: Appendix D, §1, §3.1, §4.2.
• J. Wang, W. Zhang, W. Shi, Y. Li, and J. Cheng (2026b) TCOD: exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. External Links: 2604.24005, Link Cited by: §4.2.
• L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei (2022) Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. Cited by: Appendix C, §3.1.
• Y. Wen, Z. Li, W. Du, and L. Mou (2023) F-divergence minimization for sequence-level knowledge distillation. External Links: 2307.15190, Link Cited by: §4.2.
• P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, Z. Zheng, C. Xie, and H. Yao (2026) SkillRL: evolving agents via recursive skill-augmented reinforcement learning. External Links: 2602.08234, Link Cited by: §2.2, §3.1.
• H. Xu, Z. Wang, Z. Zhu, L. Pan, X. Chen, S. Fan, L. Chen, and K. Yu (2025) Alignment for efficient tool calling of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 17787–17803. Cited by: Appendix C.
• H. Xu, Z. Zhu, L. Pan, Z. Wang, S. Zhu, D. Ma, R. Cao, L. Chen, and K. Yu (2024) Reducing tool hallucination via reliability alignment. arXiv preprint arXiv:2412.04141. Cited by: Appendix C.
• Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard (2026) TIP: token importance in on-policy distillation. External Links: 2604.14084, Link Cited by: §4.2.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1.
• C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan (2026a) Self-distilled rlvr. External Links: 2604.03128, Link Cited by: Appendix D, §3.1, §4.2.
• W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin (2026b) Learning beyond teacher: generalized on-policy distillation with reward extrapolation. External Links: 2602.12125, Link Cited by: §1.
• Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380. Cited by: Appendix C, §3.1.
• S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022) Webshop: towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems 35, pp. 20744–20757. Cited by: Appendix C, §1, §3.1, §4.1.
• T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei (2026a) On-policy context distillation for language models. External Links: 2602.12275, Link Cited by: §1, §4.2.
• Z. Ye, W. Shi, Y. Liu, Y. Wang, Z. Cai, Y. Shi, Q. Gu, X. Cai, and F. Feng (2026b) Look before you leap: autonomous exploration for llm agents. arXiv preprint arXiv:2605.16143. Cited by: Appendix C.
• Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §1, §3.4, §4.1.
• Y. Zhang, X. Lin, and C. Wu (2026) StepOPSD: step-aware online preference distillation for agent reinforcement learning. External Links: 2605.27140, Link Cited by: Appendix D, §1, §3.1, §4.2.
• S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. External Links: 2601.18734, Link Cited by: Appendix D, §1, §2.2, §3.1, §4.2.
• H. Zhou, Y. Chen, S. Guo, X. Yan, K. H. Lee, Z. Wang, K. Y. Lee, G. Zhang, K. Shao, L. Yang, et al. (2025) Memento: fine-tuning llm agents without fine-tuning llms. arXiv preprint arXiv:2508.16153. Cited by: Appendix C.

Appendix A Theoretical Analysis

A.1 From the Bayes factor to the self-teacher contrast

AgentOPSD approximates the ideal per-turn Bayes factor

${ℬ}_k=\log \frac{p\left(a_k\mid s_k,C\right)}{p\left(a_k\mid s_k,\neg C\right)}=\mathrm{logit}p\left(C\mid s_k,a_k\right)-\mathrm{logit}p\left(C\mid s_k\right)$

(13)

by the self-teacher contrast

$e_k=\log \frac{{\pi }_{\theta }\left(a_k\mid s_k,c^{+}\right)}{{\pi }_{\theta }\left(a_k\mid s_k\right)},$

(14)

where $C$ denotes eventual success and ${\rho }_k=p\left(C\mid s_k\right)$. We use two assumptions: (A1) the skill-conditioned branch is success-conditional, ${\pi }_{\theta }\left(a_k\mid s_k,c^{+}\right)\approx p\left(a_k\mid s_k,C\right)$; (A2) when success is rare (${\rho }_k$ small) the marginal is failure-dominated, ${\pi }_{\theta }\left(a_k\mid s_k\right)\approx p\left(a_k\mid s_k,\neg C\right)$.

The marginal action distribution is the success/failure mixture

${\pi }_{\theta }\left(a_k\mid s_k\right)={\rho }_{k}p\left(a_k\mid s_k,C\right)+\left(1-{\rho }_k\right)p\left(a_k\mid s_k,\neg C\right).$

(15)

Substituting (15) into (14) under (A1),

$e_k\approx {ℬ}_k-\log \left(1-{\rho }_k+{\rho }_k{e}^{{ℬ}_k}\right)\underset{{\rho }_k\to 0}{\overset{}{\to }}{ℬ}_k,$

(16)

so (A2) is the ${\rho }_k\to 0$ limit in which the contrast recovers the Bayes factor. Under (A1) alone, $e_k$ is the pointwise mutual information

$e_k\approx \log \frac{p\left(a_k\mid s_k,C\right)}{p\left(a_k\mid s_k\right)}=\log \frac{p\left(C\mid s_k,a_k\right)}{p\left(C\mid s_k\right)},$

(17)

positive iff $a_k$ raises the posterior success probability. The correction in (16) is monotone in ${ℬ}_k$, hence $\mathrm{sign}\left(e_k\right)=\mathrm{sign}\left({ℬ}_k\right)$ and $e_k$ preserves the ranking of turns by evidential strength; AgentOPSD uses $e_k$ only through this sign and ranking.

A.2 Properties of the reshaping

Let $A^{\left(i\right)}$ be the group-relative advantage, $\Delta B_k=B_k-B_{k-1}$ the per-turn belief revision, $z_k$ its within-trajectory standardization, $m_k=\mathrm{clip}(1+b\mathrm{sign}\left(A^{\left(i\right)}\right)z_k,\mathrm{\hspace{0.17em}1}-b,\mathrm{\hspace{0.17em}1}+b)$ with $b\in (0,1)$, and ${\tilde{A}}_k=A^{\left(i\right)}\left(\left(1-\lambda \right)+\lambda m_k\right)$ with $\lambda \in [0,1]$.

Proposition 1(Boundedness).

$\left|{\tilde{A}}_k-A^{\left(i\right)}\right|\le \lambda b\left|A^{\left(i\right)}\right|$, hence $\left(1-\lambda b\right)\left|A^{\left(i\right)}\right|\le \left|{\tilde{A}}_k\right|\le \left(1+\lambda b\right)\left|A^{\left(i\right)}\right|$.

Proof.

$m_k\in [1-b,1+b]$ gives $\left|m_k-1\right|\le b$, and ${\tilde{A}}_k-A^{\left(i\right)}=A^{\left(i\right)}\lambda \left(m_k-1\right)$. ∎

Proposition 2(Sign Preservation).

$\mathrm{sign}\left({\tilde{A}}_k\right)=\mathrm{sign}\left(A^{\left(i\right)}\right)$ for every turn $k$.

Proof.

$\left(1-\lambda \right)+\lambda m_k\ge 1-\lambda b>0$ since $\lambda \le 1,b<1$; a strictly positive factor preserves sign. ∎

Proposition 3(Recovery of GRPO).

At $\lambda =0$, ${\tilde{A}}_k=A^{\left(i\right)}$ for every token and the AgentOPSD gradient equals the GRPO gradient.

Proof.

$\lambda =0$ gives $\left(1-\lambda \right)+\lambda m_k=1$, so ${\tilde{A}}_k=A^{\left(i\right)}$ identically, independent of the belief signal. ∎

Proposition 4(First-Order Decomposition of the Belief Revision).

For $c_k=\gamma c_{k-1}+e_k$, ${\ell }_k=\mathrm{logit}\left(B_0\right)+c_k$, and $\Delta {\ell }_k=e_k-\left(1-\gamma \right)c_{k-1}$,

$\Delta B_k=B_{k-1}\left(1-B_{k-1}\right)\Delta {\ell }_k+O\left({\left(\Delta {\ell }_k\right)}^2\right).$

(18)

Proof.

$B_k=\sigma \left({\ell }_k\right)$ with ${\sigma }^{\prime }=\sigma \left(1-\sigma \right)$; a first-order expansion around ${\ell }_{k-1}$ gives $B_k=B_{k-1}+B_{k-1}\left(1-B_{k-1}\right)\Delta {\ell }_k+O\left({\left(\Delta {\ell }_k\right)}^2\right)$. The gate $B\left(1-B\right)$ is maximal at $B=\frac{1}{2}$ and vanishes as $B\to \{0,1\}$. ∎

Proposition 5(Exact Budget of the Idealized Recursion).

The idealized increments telescope to the endpoint change, ${\sum }_k\Delta B_k=B_K-B_0$.

Proof.

${\sum }_{k=1}^K\left(B_k-B_{k-1}\right)=B_K-B_0$. ∎

Proposition 6(Non-Identifiability of Per-Turn Contribution).

There exist two trajectories with identical outcome reward—hence identical broadcast advantage—whose per-turn contributions differ; per-turn credit is not identifiable from the trajectory return alone.

Proof (by construction).

Take ${\tau }_1,{\tau }_2$ in one group with $R\left({\tau }_1\right)=R\left({\tau }_2\right)$, so $A\left({\tau }_1\right)=A\left({\tau }_2\right)$ and GRPO assigns the same scalar to every turn. Let ${\tau }_1$ succeed through a single decisive turn ($\Delta B$ concentrated) and ${\tau }_2$ through evenly spread progress. The returns coincide but the per-turn contributions differ, so an additional per-turn signal is required to recover them. ∎

Proposition 7($B_0$ as the Group Success-Rate Estimate).

For a task with success probability ${\theta }_x$ and a group of $G$ trajectories yielding $S$ successes under a binary reward, the maximum-likelihood estimate of ${\theta }_x$ is the group success fraction $\overline{R}=S/G$, which is the standard GRPO group mean; AgentOPSD sets $B_0=\mathrm{clip}(\overline{R},{ϵ}_0,1-{ϵ}_0)$.

Proof.

Under a Binomial$(G,{\theta }_x)$ likelihood the MLE is $S/G$. The clip (${ϵ}_0={10}^{-4}$) only keeps $\mathrm{logit}\left(B_0\right)$ finite for all-correct or all-wrong groups, which have zero group-relative advantage and hence do not contribute to the update. ∎

Appendix B Algorithm

We give pseudocode for one AgentOPSD training iteration at turn-level granularity in Algorithm 1. The only addition over GRPO is a single teacher forward pass per turn and the per-turn belief reshaping block; everything else is the standard group-relative update.

媒体内容 · 前往原文查看

1:policy ${\pi }_{\theta }$, verifier $R$, group size $G$, skill retriever; mixing $\lambda$, bound $b$, evidence decay $\gamma$

2:for each training iteration do

3:  sample a batch of tasks $\left\{x\right\}$

4:  for each task $x$ with retrieved skill $c^{+}$ do

5:   sample $G$ trajectories $\{y^{\left(1\right)},\dots ,y^{\left(G\right)}\}\sim {\pi }_{\theta }(\cdot \mid x)$; trajectory $i$ has $K_i$ turns $⊳$ on-policy rollout

6:   for $i=1,\dots ,G$ do

7:     obtain reward $R^{\left(i\right)}=R(x,y^{\left(i\right)})\in \{0,1\}$ from the verifier

8:   end for

9:   $A_{\mathrm{seq}}^{\left(i\right)}\leftarrow \left(R^{\left(i\right)}-{\mu }_G\right)/{\sigma }_G$ $⊳$ group-relative advantage

10:   for $i=1,\dots ,G$ do

11:     $B_0\leftarrow \mathrm{clip}(\overline{R},{ϵ}_B,1-{ϵ}_B)$; ${\ell }_0\leftarrow \mathrm{logit}\left(B_0\right)$ $⊳$ standard GRPO group success rate $\overline{R}=S/G$ (Prop. 7)

12:     $c_0\leftarrow 0$

13:     for $k=1,\dots ,K_i$ do $⊳$ per-turn belief state

14:      $e_k\leftarrow {\sum }_t\mathrm{sg}\left[\log {\pi }_{\theta }\left(y_{k,t}\mid s_k^{+}\right)-\log {\pi }_{\theta }\left(y_{k,t}\mid s_k\right)\right]$ $⊳$ one extra teacher forward

15:      $c_k\leftarrow \gamma c_{k-1}+e_k$; ${\ell }_k\leftarrow {\ell }_0+c_k$; $B_k\leftarrow \sigma \left({\ell }_k\right)$; $\Delta B_k\leftarrow B_k-B_{k-1}$

16:     end for

17:     $q_k\leftarrow \mathrm{sign}\left(A_{\mathrm{seq}}^{\left(i\right)}\right)\Delta B_k$ for all $k$ $⊳$ outcome-aligned credit

18:     $z_k\leftarrow \left(q_k-\mathrm{mean}\left(q\right)\right)/\left(\mathrm{std}\left(q\right)+ϵ\right)$ $⊳$ within-trajectory standardization

19:     $w_k\leftarrow \mathrm{clip}(1+b{z}_k,\mathrm{\hspace{0.17em}1}-b,\mathrm{\hspace{0.17em}1}+b)$ $⊳$ bounded multiplier

20:     ${\tilde{A}}_k^{\left(i\right)}\leftarrow A_{\mathrm{seq}}^{\left(i\right)}\left[\left(1-\lambda \right)+\lambda w_k\right]$; each token inherits $\tilde{A}$ of its turn

21:   end for

22:  end for

23:  update $\theta$ by maximizing the clipped GRPO objective ${ℒ}_{\text{AgentOPSD}}\left(\theta \right)$ with $\left\{\tilde{A}\right\}$ $⊳$ policy update

24:end for

Token-level variant.

Replace the per-turn recursion with the same recursion over the flattened token sequence under a response mask: accumulate ${\delta }_t$ directly, standardize $\Delta B_t$ over the trajectory’s tokens, and assign ${\tilde{A}}_t$ per token. This is the granularity ablation reported in Table 2.

Cost.

The overhead over GRPO is one teacher forward pass per trajectory; the belief reshaping block is elementwise and adds no rollouts and no learned parameters.

Appendix C Datasets

Our experiments span three multi-turn agentic environments covering embodied household reasoning, web navigation, and search-augmented question answering.

ALFWorld

(Shridhar et al., 2020) is a text-based embodied environment with six task categories—Pick and Place, Look at Object in Light, Pick Clean then Place, Pick Heat then Place, Pick Cool then Place, and Pick Two and Place. Given a language goal and textual observations, the agent selects admissible actions until the goal is satisfied.

WebShop

(Yao et al., 2022) is an interactive online-shopping environment. For each user request the agent searches the product catalog, inspects candidate items, selects the required attributes, and attempts a purchase satisfying the specified constraints. We evaluate on the $128$ fixed validation tasks of Feng et al. (2025).

Search-QA

follows the Search-R1 setup (Jin et al., 2025) over seven datasets: single-hop NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023) and multi-hop HotpotQA (Yang et al., 2018), 2Wiki (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), Bamboogle (Press et al., 2023), with NQ and HotpotQA in-domain and the rest held out. The agent issues search queries, inspects retrieved documents (retrieval via E5 (Wang et al., 2022)), and synthesizes the collected evidence before returning its final answer (Ye et al., 2026b; Chen et al., 2026b; a; Ji et al., 2026; Zhou et al., 2025; Xu et al., 2024; 2025).

Appendix D Baseline Details

We compare against three groups of baselines. Unless a method is marked with $\ast$, evaluation uses only the standard task prompt and the interaction history returned by the environment; $\ast$ indicates that a retrieved skill is additionally supplied during validation and testing.

Vanilla.

The instruction-tuned backbone evaluated without any post-training.

Skill-Prompt^∗.

The same frozen parameters as Vanilla, but a retrieved task-relevant skill is prepended to the context at validation/test time, measuring the inference-time value of skills without any parameter update.

GRPO

(Shao et al., 2024). A critic-free group-relative RL algorithm: it samples a group of trajectories per task, normalizes their terminal rewards into relative advantages, and optimizes a clipped surrogate objective; every token inherits its trajectory’s sequence-level advantage.

Skill-GRPO / Skill-GRPO^∗.

GRPO with a retrieved skill injected into the training prompt. The skill is removed at inference for Skill-GRPO (testing whether the guidance has been internalized), and kept at inference for Skill-GRPO^∗.

OPSD

(Zhao et al., 2026). On-policy self-distillation: a teacher branch conditioned on training-only privileged context re-scores the student’s sampled tokens and produces dense token-level targets through distribution matching; the teacher outputs are detached and the privileged context is not used at inference.

GRPO+OPSD.

Jointly optimizes the trajectory-level GRPO loss and the token-level OPSD objective, a straightforward combination of outcome-based RL and generic self-distillation.

Skill-SD

(Wang et al., 2026a). Supplies the retrieved skill only to the teacher branch and trains the student to absorb the skill-conditioned guidance via an importance-weighted distillation loss, without requiring skills at evaluation.

RLSD

(Yang et al., 2026a). Converts the teacher–student log-probability gap into a bounded coefficient that scales the magnitude of each token’s GRPO update; the sign of the update remains determined by the outcome-derived advantage.

SDAR

(Lu et al., 2026b). Adds a separately gated auxiliary self-distillation loss on top of GRPO, leaving the original GRPO advantage unchanged and using a bounded gate to modulate each teacher signal.

StepOPSD

(Zhang et al., 2026). Applies the teacher–student self-distillation signal at the turn (step) level rather than per token, but uses each step’s local signal in isolation.

All post-training baselines share the same backbone models, environment interfaces, data, and training budget as AgentOPSD; they differ primarily in their optimization objective and in whether skills are available during training or evaluation.

Appendix E Evaluation Metrics

ALFWorld.

We report the overall success rate over the evaluation tasks (the fraction of episodes that reach the specified goal), which is the sample-weighted average of the six per-category success rates.

Search-QA.

We report the overall accuracy over all evaluation questions aggregated across the seven datasets.

WebShop.

We report a normalized completion Score (averaged over partial-constraint satisfaction and scaled by $100$) and an exact-completion success rate Succ. (the percentage of episodes that satisfy all specified requirements).

Appendix F Hyperparameters

Table 3 summarizes the hyperparameters used by AgentOPSD across all our experiments. We deliberately use a single setting for every environment and model scale rather than tuning per task: AgentOPSD runs at turn-level granularity with reshaping weight $\lambda =0.5$, multiplier band $b=0.2$, gap accumulation with decay $\gamma =0.95$, policy clipping ${ϵ}_{\mathrm{low}}=0.2$ / ${ϵ}_{\mathrm{high}}=0.24$, and the empirical group success rate $\overline{R}=S/G$ (clipped) as the state prior $B_0$. The sensitivity study in §3.4 sweeps $\lambda$, $\gamma$ and ${ϵ}_{\mathrm{high}}$ around this setting and finds no swept value that improves on it by a meaningful margin, which is why one shared configuration is used throughout rather than per-environment tuning.

媒体内容 · 前往原文查看

Method	$\eta$	$G$	${ϵ}_{\mathrm{low}}/{ϵ}_{\mathrm{high}}$	${\alpha }_{\mathrm{KL}}$	$\lambda$	$b$	$\gamma$	SRS
AgentOPSD	${10}^{-6}$	8	0.2/0.24	0.01	0.5	0.2	0.95	KM

We use a single shared optimization recipe across environments (learning rate, group size, PPO clip range, and KL coefficient in Table 3; dual-clip constant $c=3.0$, gradient clipping $1.0$, entropy coefficient $0.001$, one PPO epoch per update, and FSDP on a single node). The remaining settings are environment-specific and summarized in Table 4.

媒体内容 · 前往原文查看

	ALFWorld	WebShop	Search-QA
Training steps	150	150	150
Train batch size	16	16	128
Rollout group size $G$	8	8	8
Max prompt length	2048	4096	4096
Max response length	512	512	512
Max interaction turns	50	15	4
Rollout temperature (train / val)	1.0 / 0.4	1.0 / 0.4	1.0 / 0.4
GPUs (tensor-parallel size)	8 (2)	2 (2)	4 (1)

Appendix G Training Dynamics

We present the full training dynamics of AgentOPSD across all model scales and environments in Figures 4–5, tracking the teacher–student gap and the reward throughout training.

Appendix H Prompt

Figures 6–8 present the full prompt templates used by AgentOPSD for the three evaluation environments, where {skill_context} is populated with the retrieved skill during training and left empty at inference time.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看