摘要
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的一种可扩展范式。然而，其有效性从根本上受到探索过程的限制：策略只能改进其已经采样过的轨迹。虽然增加 rollout 数量可以缓解这一问题，但这种暴力扩展方式计算成本高昂，而现有修改优化目标的方法对探索内容的控制能力有限。在这项工作中，我们提出了 NudgeRL，一种用于 RLVR 中结构化且多样性驱动的探索框架。我们的方法引入了策略引导（Strategy Nudging），该方法将每次 rollout 的条件设定为轻量级的策略级上下文，从而在不依赖昂贵的神谕监督的情况下，诱导出多样化的推理轨迹。为了有效利用这种结构化探索进行学习，我们进一步提出了一个统一目标，该目标将奖励信号分解为上下文间和上下文内两个组成部分，并引入一个蒸馏目标，将发现的行为迁移回基础策略。实验表明，NudgeRL 在性能上优于使用高达 8 倍更大 rollout 预算的标准 GRPO，同时在五个具有挑战性的数学基准测试中，平均表现优于基于神谕的强化学习基线。这些结果表明，结构化的、由上下文驱动的探索可以作为一种高效且可扩展的替代方案，既替代了暴力 rollout 扩展，也替代了基于特权信息的可行性导向方法。我们的代码可在 https://github.com/tally0818/NudgeRL 获取。
1 引言
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型（LLM）推理能力的一种强大范式 [20, 7]。通过利用可验证奖励，诸如群体相对策略优化（GRPO）[18] 等方法无需密集监督即可实现可扩展的后训练。该范式已成功应用于广泛的领域。
尽管 RLVR 取得了成功，但其探索推理轨迹空间的能力从根本上限制了它的发展。一种自然的做法是增加采样 rollout 的数量，这能提高发现罕见轨迹的概率 [5]。然而，这种蛮力式的规模扩展很快就会在计算上变得难以承受，从而促使人们探索能够提高探索效率的替代方法。
近期的工作试图通过修改优化目标来解决这一局限，例如采用熵正则化或解耦裁剪 [26, 24]。虽然这些方法在分布层面鼓励更广泛的探索，但它们对探索内容的控制有限，并且常常无法确保覆盖具有语义意义的推理策略。另一条研究路线利用特权信息，例如 oracle 解决方案或中间推理步骤，来提高发现正确轨迹的可行性 [27, 16, 8, 19]。尽管这些方法有效，但它们主要面向可行性，并且依赖于获取成本高昂且难以扩展的强监督信号。此外，通过将策略引导向一组狭窄的、预定义的成功轨迹，它们可能会限制探索的多样性，并阻碍发现替代性的推理策略 [25, 23]。
在本工作中，我们通过以可扩展的方式显式构建推理空间的结构，来解决探索瓶颈问题。我们提出了 NudgeRL 框架，该框架在探索阶段引入了策略提示（Strategy Nudging）。策略提示不依赖昂贵的 oracle 数据，而是在输入中附加轻量级的启发式文本提示（例如，针对数学问题的特定策略或推理关键词）。这刻意迫使模型遍历那些在纯粹朴素采样下可能被忽略的、不同且多样的推理模式。
然而，从这种基于上下文条件的探索中学习会带来新的挑战。由于轨迹是在不同的上下文条件提示下生成的，样本自然地被划分为多个不同的组，其中奖励的变化既反映了轨迹本身的质量，也反映了特定上下文的偏差，这使得标准的组间优势估计变得不可靠。此外，上下文强制（context forcing）造成了轨迹采样方式与策略最终在推理时使用方式之间的不匹配。如果不进行干预，在上下文强制探索下发现的改进可能无法直接迁移到基础策略上。为应对这些挑战，我们进一步引入了：(i) 组间-组内优势（Inter-Intra group advantage），以实现跨上下文诱导组的有效信用分配；以及 (ii) 一个蒸馏增强目标，该目标显式地将上下文强制探索期间发现的有效行为迁移回基础策略。
我们的方法实现了结构化和多样性驱动的探索，同时与标准的 RLVR 流程完全兼容。实验表明，即使 GRPO 拥有更大的轨迹生成预算，NudgeRL 的性能也能超越 GRPO，并且优于 oracle 引导的基线方法。这表明，可扩展的、面向多样性的探索可以作为一种有效的替代方案，既能替代暴力式的轨迹扩展，也能替代依赖可行性的特权信息。
2 预备知识
2.1 组相对策略优化（GRPO）
我们考虑一个提示词的实证分布。对于每个提示词，一个策略会生成一组采样轨迹，其中每条采样轨迹按如下方式采样。每条采样轨迹由一个可验证的奖励函数进行评估。
与通常使用学习到的价值函数来估计优势值的标准 PPO [17] 不同，GRPO [18] 从组内奖励中推导出优势值。对于从同一提示词采样的采样轨迹，令 表示采样轨迹 的奖励值。组内优势值定义如下：
| (1) |
其中 和 分别是该组内奖励的均值和标准差， 用于数值稳定性。这样可以在不训练价值函数的情况下得到相对优势估计。
然后，策略通过一个 PPO 风格的裁剪目标函数进行优化：
| (2) |
因此，GRPO 保留了 PPO 的裁剪目标函数，同时使用了组内相对优势。
2.2 动机：从探索到性能提升
为了理解为什么探索是 RLVR 中的一个根本瓶颈，我们超越轨迹级别的奖励，考察生成 token 的概率质量在训练过程中如何变化。Hu 等人 [5] 将 RLVR 中期望的单步性能提升 () 描述为：
| (3) |
其中 和 分别表示正确和错误 token 的总概率质量， 是学习率， 是采样轨迹数量。 和 是已采样正确和错误 token 的二阶矩，而 和 是未采样正确和错误 token 的二阶矩。 表示来自已采样 token 的净奖励贡献。
由于 ，公式 3 中的前两项为非负项，推动学习向前进行。然而，第三项则是一个潜在的惩罚项。因为错误 token 通常主导概率质量（），一个较大的 ，意味着模型在正确轨迹上拥有显著的概率质量但未能成功探索，这会产生一个主导性的负向力，阻碍性能提升。因此，RLVR 的核心瓶颈在于未被探索的正确区域。
采样轨迹扩展的局限性。
为了减轻这一惩罚，一个简单的解决方案是增加采样轨迹数量 。Hu 等人 [5] 表明，对于一组概率为 的 token，经过 次抽取后，期望的未采样二阶矩为：
| (4) |
该值随 单调递减。然而，概率较小的 token 衰减缓慢，因此要完全覆盖长尾的正确轨迹，需要极其庞大的 rollout 预算。
这凸显了盲目扩大规模以减少未探索正确质量（）的局限性。即使在大规模 下，长尾正确轨迹仍难以被采样，这表明需要一种结构化探索机制，能够高效地暴露此类潜在轨迹。
3 NudgeRL
我们提出 NudgeRL，一种用于 RLVR 中结构化探索与学习的框架。NudgeRL 包含三个组成部分：(i) 策略引导（Strategy Nudging），通过在策略级上下文条件下生成 rollout，以诱导多样化的推理轨迹；(ii) 组内-组间优势（Inter-intra Group Advantage），一种信用分配方法，能够实现对策略的受控探索与利用；(iii) 蒸馏增强的 RL 目标函数，用于从上下文条件化的 rollout 中学习，并将有效策略蒸馏到原始提示词下的策略中，以便在无外部上下文的情况下进行推理。
3.1 策略引导：通过策略级上下文实现结构化探索
鉴于先前工作 [5] 通过更大的 rollout 预算减少未采样概率质量来缓解探索瓶颈，一个自然的问题随之产生：需要多少次 rollout 才能可靠地发现一条稀有轨迹？为了量化这一发现成本，考虑一条概率为 的稀有轨迹。至少观察到一次所需的预期 rollout 次数为：
| (5) |
这意味着对于低概率轨迹，所需的 rollout 预算会变得极其庞大。在实践中，朴素的 rollout 扩大会反复从当前策略的高概率模式中采样，导致覆盖稀有轨迹的收益递减。
这启发我们，在能够将采样分布转向原本稀有轨迹的上下文条件下进行生成。如果这样的上下文能提高轨迹 的概率，即 ，那么其预期 rollout 次数变为：
| (6) |
因此，上下文无需提供解决方案；它们可以作为轻量级的控制手段，改变采样分布并降低发现稀有轨迹的成本。
策略引导。
尽管上下文条件化在原则上能够提升探索效率，但简单地将多个上下文放入同一条提示词中，会将策略的选择权交给模型策略本身，这可能导致模型忽略部分上下文，并反复遵循主导性的推理模式。为了强制覆盖所有上下文，我们在每次生成前，为每一次 rollout 分配一个单一的采样上下文。
设 \( \mathcal{C} \) 为原始提示词 \( p \) 的一个策略级上下文池。对于每个 rollout 索引 \( i \)，我们首先从池中采样 \( c_i \sim \mathcal{C} \)。为了避免完全依赖上下文池，并保持与原始提示词的兼容性，我们进一步应用上下文丢弃（context dropout）。具体来说，我们采样一个掩码 \( m_i \sim \text{Bernoulli}(\rho) \)，并将上下文定义为：
| (7) |
然后，我们构建最终的提示词 \( p_i = [c_i'; p] \)，并生成 \( y_i \sim \pi_{\theta}(\cdot | p_i) \)。通过在不同 rollout 索引间改变 \( c_i' \)，策略引导在输入条件化层面引入了多样性，而非仅仅依赖于从单一提示词中进行采样。关于生成 \( \mathcal{C} \) 的细节见附录 B。
上下文引发的 rollout 多样性。
为了验证策略引导是否确实带来了预期的多样性，我们将其与无上下文条件化的朴素采样方法进行了比较。对于每个提示词，两种方法总共生成 8 个 rollout：策略引导方法从 2 个上下文中各采样 4 个 rollout（不使用上下文丢弃），而基线方法则在原始提示词下，从基础策略中采样全部 8 个 rollout。随后，我们使用大语言模型作为评判者（LLM-as-a-judge，gpt-4o-mini [15]）对推理结构进行聚类，并测量不同聚类的数量；更多细节见附录 B。
如图 1 所示，与朴素采样相比，策略引导更频繁地增加了不同推理结构的数量，而基础策略则常常收敛到相似的模式。这表明，策略引导在进行任何策略更新之前，就已经使探索过程多样化，从而允许在相同的 rollout 预算下，让 rollout 集合覆盖更广泛的推理模式。
3.2 组间-组内优势：学习在策略间平衡探索
GRPO 通过比较同一提示词分布下各 rollout 之间的奖励来估计优势。然而，在使用策略引导时，rollout 是从上下文条件提示词中抽取的。因此，单一组基线会混淆不同上下文引起的奖励变化，从而扭曲分配给每个 rollout 的相对优势。
为解决这一问题，我们提出了组间-组内优势，通过两个互补信号进行信用分配：一个组内信号，用于捕捉相同条件上下文下轨迹的质量；一个组间信号，用于捕捉上下文本身的相对可靠性。
给定采样的 rollout 及其奖励，我们根据它们被分配的上下文进行分组。上下文组的集合定义为
| (8) |
对于每个组 ，我们定义索引集 ，该集合划分了所有 rollout。然后我们计算上下文级别和全局的奖励基线：
| (9) |
利用这些基线，我们将优势定义为：
| (10) |
其中 和 分别是 的均值和标准差， 用于保证数值稳定性。
由于优势决定了策略更新的方向，它们应与底层奖励保持一致，同时允许上下文级别的偏好影响信用分配。
命题 3.1。
考虑来自上下文组 和 的两条轨迹 和 ，其奖励分别为 和 。设 和 为对应的上下文均值， 和 为它们的优势。在二元奖励设定下，如果 ，那么：
| (11) |
因此，对于 ，更高的奖励始终对应更高的优势值，从而确保与底层目标的一致性；上下文仅影响等奖励轨迹之间的相对排序。对于等奖励轨迹， 控制上下文层面的偏好： 倾向于来自较低奖励上下文的成功轨迹，鼓励探索非典型上下文；而 则倾向于来自较高奖励上下文的成功轨迹，强调更可靠的上下文。中性情况 对不同上下文中的等奖励轨迹一视同仁； 的情况如图 2 (a) 所示。
3.3 训练目标
尽管策略引导通过从上下文条件提示词 中采样轨迹来改进探索，但推理时的目标策略应在没有外部上下文的情况下运行。因此，在 下发现的有用轨迹必须迁移到基础策略 上。
为弥合这一差距，我们引入了 Song 等人 [19] 提出的优势加权蒸馏项，该方法直接使用在上下文条件输入 下采样的轨迹来更新策略：
| (12) |
与标准行为克隆不同，该公式选择性地强调具有高归一化优势的轨迹，确保只有在不同上下文下发现的有用行为才能贡献给 的更新。
同时，我们在上下文条件策略上优化强化学习目标：
| (13) |
最终目标结合了这两项：
| (14) |
该目标引入了一种互补的学习动态。RL 项作用于上下文条件策略，改进探索并强化每个上下文内的成功轨迹。相比之下，蒸馏项将这些改进投射到基础提示词策略上，从而实现跨上下文的泛化。因此，模型学会了在推理时无需依赖显式上下文即可复现有效的推理策略。与公式 2 中的 GRPO（其在原始提示词 下采样并优化轨迹）不同，NudgeRL 在 下对上下文条件轨迹进行 RL，同时将高优势轨迹蒸馏回基础策略 中。
4 实验
| 模型 | 方法 | Rollouts () | AIME24 | AIME25 | AMC23 | MATH500 | APEX | 平均 |
|---|---|---|---|---|---|---|---|---|
| Qwen3-4B- Instruct | 基础模型 | – | 0.374 | 0.352 | 0.653 | 0.592 | 0.036 | 0.402 |
| GRPO | 8 | 0.444 | 0.367 | 0.749 | 0.668 | 0.040 | 0.454 | |
| 16 | 0.454 | 0.355 | 0.840 | 0.655 | 0.045 | 0.470 | ||
| 32 | 0.451 | 0.370 | 0.881 | 0.674 | 0.058 | 0.487 | ||
| 64 | 0.415 | 0.324 | 0.848 | 0.641 | 0.027 | 0.451 | ||
| POPE†[16] | 8 | 0.460 | 0.337 | 0.838 | 0.652 | 0.048 | 0.467 | |
| \rowcolorciteblue!10 \cellcolorwhite | NudgeRL | 8 | 0.482 | 0.393 | 0.857 | 0.660 | 0.053 | 0.489 |
| Olmo3-7B- Instruct-SFT | 基础模型 | – | 0.134 | 0.118 | 0.467 | 0.384 | 0.021 | 0.225 |
| GRPO | 8 | 0.187 | 0.159 | 0.537 | 0.434 | 0.025 | 0.268 | |
| 16 | 0.188 | 0.176 | 0.548 | 0.461 | 0.023 | 0.279 | ||
| 32 | 0.195 | 0.176 | 0.553 | 0.459 | 0.024 | 0.281 | ||
| 64 | 0.081 | 0.053 | 0.349 | 0.291 | 0.027 | 0.160 | ||
| POPE†[16] | 8 | 0.186 | 0.169 | 0.558 | 0.460 | 0.023 | 0.279 | |
| \rowcolorciteblue!10 \cellcolorwhite | NudgeRL | 8 | 0.190 | 0.179 | 0.563 | 0.468 | 0.025 | 0.285 |
4.1 实验设置
基线方法。
我们将我们的方法与以下方法进行比较：(i) 未经优化的基础模型，作为参考点；(ii) 使用递增 rollout 预算的 GRPO，其中，这评估了将朴素 rollout 扩展作为一种暴力探索策略的效果；(iii) POPE [16]，该方法通过在基础提示词末尾附加 oracle 解决方案的前缀来增强标准 GRPO，从而缓解稀疏奖励信号的瓶颈。更多细节见附录 C。
评估数据集与指标。
AIME24 和 AIME25，各含 30 道题的奥林匹克风格高中竞赛 [13]；AMC23，一个含 40 道题的高中竞赛基准 [12]；MATH500 的 level-5 子集，包含 134 道高难度数学题 [4]；以及 Apex Shortlist，包含 48 道高级竞赛风格问题 [1]。我们报告 pass@1，该指标使用 Chen 等人 [2] 的无偏估计器，从 128 次 rollout 中估计得出。所有解答均使用 math-verify [6] 自动评分。更多细节见附录 E。
实现细节。
我们将 NudgeRL 应用于 Qwen3-4B-Instruct-2507 [21] 和 Olmo-3-7B-Instruct-SFT [14]，并使用 DAPO-17k-Processed 作为训练集 [24]。为了构建上下文池，我们使用 gpt-4o-mini [15] 为每个问题（例如勾股定理）生成两个策略级上下文，并在未进行额外验证的情况下直接使用它们（即 ）。对于 POPE 基线，使用 DeepSeek Reasoner v3.2 [9] 生成了 oracle 解决方案。我们在附录 D 中提供了额外的优化细节。
4.2 主要结果
NudgeRL 以更少的 rollout 次数达到了更大预算 GRPO 的效果。
如表 1 所示，NudgeRL 在两个模型上均取得了最佳平均性能，同时每个提示词仅使用 8 次 rollout。在 Qwen3-4B-Instruct-2507 上，NudgeRL 达到了 0.489 的平均 pass@1，略优于 GRPO 在 32 次 rollout 时的最佳结果（0.487），并以 8 倍更小的 rollout 预算超越了 GRPO 在 64 次 rollout 时的结果（0.451）。在 Olmo3-7B-Instruct-SFT 上，NudgeRL 同样优于 GRPO 的最佳结果，达到了 0.285，而 GRPO 在 32 次 rollout 时为 0.281。这些结果表明，仅靠更大的 rollout 预算是不够的：GRPO 在两个模型上均先提升后下降，表明在暴力 rollout 扩展下存在不稳定性。相比之下，NudgeRL 通过策略引导（Strategy Nudging）提高了探索质量，而非依赖更多的采样 rollout，从而实现了更强的性能。
与 oracle 前缀方法的比较。
我们还与 POPE [16] 进行了比较，该方法通过基于 oracle 解决方案前缀生成 rollout 来增强 GRPO。与依赖昂贵且不可扩展的 oracle 提示 [16] 或文本反馈 [19] 的基线不同，我们的方法确保了可扩展的多样性。我们使用轻量级大语言模型（例如 gpt-4o-mini）廉价地生成未经验证的策略级上下文，以诱导多种推理方向。尽管监督信号较弱，我们的方法仍持续优于 oracle 引导的基线，这表明在多样化策略上进行结构化探索比注入狭窄的特权解决方案信号更为有效。
4.3 多样化推理模式的高效覆盖
如第 3.1 节所述，仅依赖扩展 rollout 预算在发现长尾、低概率推理模式时存在严重的样本效率低下问题。这是因为朴素的 rollout 扩展会反复将计算资源分配给主导轨迹。为了实证研究策略引导（Strategy Nudging）如何克服这一探索瓶颈并提升样本效率，我们比较了 NudgeRL 与 GRPO 在逐步增大的 rollout 预算下的训练动态。我们每训练 50 步，在 AIME24 和 AIME25 联合基准上对模型进行评估，方法是对每个问题采样 64 次 rollout，并估算 pass@1 和 pass@8。
如图 3(b) 所示，NudgeRL 的提升速度快于 GRPO 变体，并且在大部分训练过程中始终是最强的方法。到第 200 步时，NudgeRL 在 AIME24/25 上的得分超过 0.42，而 GRPO 变体则保持在 0.41 左右或以下，并且随着 rollout 预算的增加，其提升速度较慢或稳定性较差。这表明策略引导通过更早地暴露有用的推理轨迹来提升样本效率，而不仅仅是增加采样的 rollout 数量。在相同的训练 rollout 预算下，增加样本数量（）进一步验证了这一趋势。如图 3(c) 所示，NudgeRL 在整个范围内始终优于 GRPO-8，这表明策略引导提升了推理时的样本效率，在达到相同 水平时所需的生成解更少。
4.4 案例研究
为了探究 NudgeRL 性能提升的来源，我们分析了一道 AIME25 题目，在该题目上，经过 NudgeRL 训练的模型成功采样到了正确的推理轨迹，而 GRPO 训练的模型则完全失败。我们采样了 32 条 rollout，并对其主要的推理策略进行了分类。
如图 4 所示，两个模型都主要依赖坐标几何方法。然而，GRPO 训练的模型还探索了一些无效策略，例如对称性假设和面积分解，这些策略始终导致截断的解，使得全部 32 条轨迹均告失败。虽然 GRPO 仅采样过一次鞋带公式策略，但 NudgeRL 显著提高了该策略的出现频率，并成功利用它生成了正确的轨迹。
这种行为凸显了我们框架的互补作用：策略引导（Strategy Nudging）暴露了罕见但有效的推理模式，例如鞋带公式策略；而组间-组内优势（Inter-Intra Group Advantage）则在发现此类可靠策略后对其进行强化和利用。详情见附录 F。
4.5 训练过程中上下文的影响
我们还报告了在使用 NudgeRL 训练 Qwen3-4B-Instruct-2507 过程中的丢弃奖励均值（dropout reward mean）和提示奖励均值（hinted reward mean）。如图 5 所示，在整个训练过程中，两种奖励都同步提升，这表明在上下文条件探索下发现的轨迹，通过蒸馏目标成功迁移到了基础策略中。有趣的是，在训练过程中，丢弃奖励偶尔会超过提示奖励。这与先前基于特权信息的可行性导向方法 [16, 27, 8, 19] 形成对比。在我们的方法中，上下文的主要作用并非直接简化问题，而是诱导出多样化的推理轨迹，这些轨迹随后可以被无上下文的策略内化吸收。
4.6 NudgeRL 的底层机制
为了进一步理解 NudgeRL 性能提升的来源，我们使用 Qwen3-4B-Instruct-2507 [21] 在一组基准测试子集上进行了一系列控制实验。
消融实验。
如图 7(a) 所示，适中的丢弃率（）在各项基准测试中始终能取得最佳性能。上下文丢弃具有双重作用：一方面，它通过偶尔回退到基础提示词，使得模型能够在固定上下文之外进行探索；另一方面，它通过更均衡的样本分布来稳定组内统计量。当丢弃率较低时，探索被限制在预定义的上下文内；而当丢弃率较高时，上下文强制的影响则会减弱。这些结果表明，在上下文条件样本和无上下文样本之间保持均衡混合，对于实现多样化的探索和稳定的优化都至关重要。
提示采样。
我们通过比较两种策略——随机采样和排名靠前选择——来研究所采样上下文的质量对性能的影响。在排名靠前的设置中，我们首先生成一个包含五个候选上下文的池子，然后针对每个问题，根据 oracle 评估的衡量标准，选出能带来最大改进的两个上下文。
如图 7(b) 所示，在指标上，随机采样的表现始终优于排名靠前的选择。虽然排名靠前的上下文能确保更高的正确性，但它们往往集中在较窄的推理策略集合上。相比之下，随机采样会在合理的推理路径上产生更广泛的分布，从而在有限的 rollout 预算下实现更有效的探索。
这些结果表明，在我们的框架内，上下文的主要作用并非提供单一的“最佳提示”，而是促进推理的多样性。因此，简单的随机采样不仅足够，而且对于可扩展且有效的基于上下文的探索来说，也是更优的选择。
通过 实现的探索与利用权衡。
图 7(a) 展示了不同 值的影响，其中 取得了最佳性能。这一趋势与我们在第 3.2 节中的命题 3.1 一致。由于策略引导已经在采样阶段确保了足够的多样性，增大 并不会阻碍跨上下文的探索。相反，它会通过优先选择来自更可靠上下文的轨迹，来强化每个问题内部的利用。这导致每个实例更一致地学习到高质量解决方案，从而解释了在 处观察到的性能提升。
蒸馏系数。
如图 7(b) 所示，移除蒸馏项（）会导致明显的性能下降，这表明将上下文发现的轨迹显式地迁移到基础策略中是至关重要的。然而，过大的 值也会降低性能，这很可能是由于策略被过度约束到采样轨迹上。适中的系数（）取得了最佳结果，表明蒸馏应当与底层的强化学习目标相辅相成。
4.7 与缩放方法的比较。
我们进一步将我们的算法与解耦裁剪 [24] 进行比较：其中 通过放大成功轨迹的贡献来控制策略更新的强度。因此，增大 允许对正优势轨迹进行更激进的策略更新。如图 7(c) 所示，在先前工作 [18, 24] 所使用的适中范围内，增大 通常会提升 GRPO 的性能。然而，我们的方法在从中等到极值的整个缩放范围内始终优于 GRPO。这表明，提升探索质量比单纯增加随机策略更新的幅度更为有效。此外，在近期 RLVR 设置 [10] 所采用的更极端缩放下，GRPO 在 处性能急剧恶化。我们认为，这种退化突显了纯随机分布层面探索的一个局限性：仅增加更新幅度对探索内容几乎无法控制。
评估的完整结果见附录 G。
5 结论
在本工作中，我们提出了 NudgeRL，一个用于 RLVR 中结构化探索的框架。我们的方法利用策略引导（Strategy Nudging），通过从轻量级的、基于策略级别的上下文条件分布中采样，来诱导多样化的推理轨迹，并通过蒸馏增强的 RL 目标函数从中学习。实验表明，与使用多达 8 倍更大 rollout 预算的 GRPO 相比，NudgeRL 取得了更优的性能，并且在不同模型上进一步超越了基于 oracle 前缀的基线方法。
局限性与未来工作
NudgeRL 的一个实际考量是生成策略级别上下文的成本。然而，这是一个在训练前一次性完成的离线过程，使用轻量级 LLM（例如 gpt-4o-mini），并且生成的上下文可以在多次训练运行中重复使用，而无需额外开销。一个更根本的局限性在于上下文的生成方式独立于正在训练的模型。上下文强制（Context Forcing）的好处源于诱导出在当前策略下不太可能出现的轨迹。然而，随着训练的进行，固定的上下文池可能会因为策略的适应而变得信息量不足。未来一个有前景的方向是模型自适应上下文生成，它能够动态构建针对当前策略盲点定制的上下文，从而可能在整个训练过程中带来更一致的探索收益。
参考文献
- [1] M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev (2025-02) MathArena: evaluating llms on uncontaminated math competitions. SRI Lab, ETH Zurich. External Links: Link Cited by: §4.1.
- [2] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §4.1.
- [3] J. Deng, J. Chen, Z. Chen, W. X. Zhao, and J. Wen (2025) Decomposing the entropy-performance exchange: the missing keys to unlocking effective reinforcement learning. arXiv preprint arXiv:2508.02260. Cited by: §A.2.
- [4] D. Hendrycks, C. Burns, S. Basart, A. Zou, D. Song, 和 J. Steinhardt (2021) 使用 MATH 数据集衡量数学问题求解能力。NeurIPS。引用于：§4.1。
- [5] J. Hu, M. Liu, X. Lu, F. Wu, Z. Harchaoui, S. Diao, Y. Choi, P. Molchanov, J. Yang, J. Kautz, 等人 (2025) Brorl：通过拓宽探索来扩展强化学习。arXiv 预印本 arXiv:2510.01180。引用于：§A.2, §1, §2.2, §2.2, §3.1。
- [6] HuggingFace (2024) Math-verify：一个用于验证数学推理的工具包。注：https://github.com/huggingface/Math-Verify 访问日期：2026-05-06 引用于：§4.1。
- [7] N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, 等人 (2024) Tulu 3：推动开放语言模型后训练的前沿。arXiv 预印本 arXiv:2411.15124。引用于：§1。
- [8] B. Liao, H. Dong, X. Xu, C. Monz, 和 J. Bian (2026) 自我提示语言模型增强强化学习。arXiv 预印本 arXiv:2602.03143。引用于：§A.3, §A.3, §1, §4.5。
- [9] A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, 等人 (2025) Deepseek-v3.2：推动开放大语言模型的前沿。arXiv 预印本 arXiv:2512.02556。引用于：§A.1, §A.1, §4.1。
- [10] M. Liu, S. Diao, X. Lu, J. Hu, X. Dong, Y. Choi, J. Kautz, 和 Y. Dong (2025) Prorl：延长强化学习扩展大语言模型的推理边界。arXiv 预印本 arXiv:2505.24864。引用于：§A.1, §4.7。
- [11] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, 和 M. Lin (2025) 理解类似 R1-zero 的训练：一个批判性视角。arXiv 2503.20783。外部链接：Link 引用于：§A.1。
- [12] 美国数学协会 (2023) 美国数学竞赛。注：https://www.maa.org/math-competitions 引用于：§4.1。
- [13] 美国数学协会 (2025) AIME：美国邀请赛数学考试。注：https://www.maa.org/math-competitions 引用于：§4.1。
- [14] T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi (2025) Olmo 3. 外部链接: 2512.13961, 链接 被引用自: §4.1.
- [15] OpenAI (2024) GPT-4o mini. 注释: https://openai.com/ko-KR/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ 访问日期: 2026-05-04 被引用自: §3.1, §4.1.
- [16] Y. Qu, A. Setlur, V. Smith, R. Salakhutdinov, and A. Kumar (2026) POPE: 通过特权在线策略探索学习解决难题的推理能力. arXiv 预印本 arXiv:2601.18779. 被引用自: §A.3, §A.3, 附录 C, 附录 C, §1, §4.1, §4.2, §4.5, 表 1, 表 1.
- [17] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) 近端策略优化算法. arXiv 预印本 arXiv:1707.06347. 被引用自: §2.1.
- [18] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: 在开放语言模型中推动数学推理的极限. arXiv 2402.03300. 外部链接: 链接 被引用自: §A.1, §A.1, §A.3, §1, §2.1, §4.7.
- [19] Y. Song, L. Chen, F. Tajwar, R. Munos, D. Pathak, J. A. Bagnell, A. Singh, and A. Zanette (2026) 通过文本反馈扩展强化学习的能力. arXiv 预印本 arXiv:2602.02482. 被引用自: §A.3, §A.3, §1, §3.3, §4.2, §4.5.
- [20] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025) Kimi k1. 5: 使用大语言模型扩展强化学习. arXiv 预印本 arXiv:2501.12599. 被引用自: §A.1, §1.
- [21] Q. Team (2025) Qwen3 技术报告。外部链接：2505.09388，链接 被引用：附录 B，§4.1，§4.6。
- [22] TRL：Transformers 强化学习 外部链接：链接 被引用：附录 D。
- [23] F. Wu, W. Xuan, X. Lu, Z. Harchaoui, 和 Y. Choi (2025) 无形的束缚：为何 RLVR 可能无法摆脱其根源。arXiv 2507.14843。外部链接：链接 被引用：§1。
- [24] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, 和 M. Wang (2025) DAPO：一个大规模开源大语言模型强化学习系统。arXiv 2503.14476。外部链接：链接 被引用：§A.1，§A.1，§A.2，附录 B，§1，§4.1，§4.7。
- [25] Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, 和 G. Huang (2025) 强化学习是否真的能激励大语言模型超越基础模型的推理能力？arXiv 2504.13837。外部链接：链接 被引用：§1。
- [26] X. Zhang, X. Yuan, D. Huang, W. You, C. Hu, J. Ruan, K. Chen, 和 X. Hu (2025) 重新发现熵正则化：自适应系数释放其在 LLM 强化学习中的潜力。arXiv 预印本 arXiv:2510.10959。被引用：§A.2，§1。
- [27] X. Zhang, Z. Huang, Y. Li, C. Ni, J. Chen, 和 S. Oymak (2025) BREAD：基于专家锚点的分支 rollout 连接 SFT 与 RL 进行推理。arXiv 预印本 arXiv:2506.17211。被引用：§A.3，§A.3，附录 C，§1，§4.5。
附录 A 相关工作
A.1 基于可验证奖励的强化学习
基于可验证奖励的强化学习（RLVR）已成为一种可扩展的范式，用于提升大语言模型的推理能力[20, 18, 24, 9]。通过利用可自动验证的信号（例如数学问题中的精确答案或代码生成中的测试用例正确性），RLVR 能够在无需密集人工监督的情况下实现有效的策略优化。
一种代表性方法是群体相对策略优化（GRPO）[18]，该方法用采样轨迹间的群体比较来替代价值函数估计，通过每组内的相对奖励差异推导优势值。在此框架基础上，后续研究引入了解耦裁剪[24]和替代归一化策略[11]等改进措施，以提升训练稳定性。
这些方法已成功应用于多种推理任务和不同规模的模型[10, 9]，使基于强化学习的价值对齐（RLVR）成为大语言模型的标准后训练方法。然而，其有效性根本上依赖于探索：策略只能改进已采样过的轨迹。因此，探索不足会直接限制学习效果，成为RLVR的关键瓶颈。接下来我们将审视现有工作如何应对这一挑战。
A.2 RLVR中的探索机制
提升探索最直接的方法是增加采样轨迹数量。已有研究表明，这种轨迹缩放可通过降低未采样区域的概率密度来显著提升性能[5]。但该方法计算成本高昂，在大规模场景下往往难以实际应用。
更常见的是，近期方法尝试通过目标函数设计来促进探索，例如熵正则化[26, 3]或解耦裁剪[24]。虽然这些方法能引导更新方向偏向探索，但无法保证训练过程中确实采样到有用或罕见的模式。换言之，调整分布形态并不一定能确保覆盖有意义的轨迹，这使得探索从根本上仍受到限制。
此外，这种分布层面的探索本质上是随机的且不受约束，可能会将策略推向语义上不理想的方向。增加熵或激进地重新加权概率，或许能鼓励模型探索低概率区域，但若缺乏结构性引导，这往往会导致产生不连贯或无效的轨迹，而非有意义的推理策略。因此，这些方法无法控制策略的探索方式，也无法提供结构化的、针对策略层面的探索，从而难以锁定多样且语义上有效的推理模式。
A.3 特权信息的使用
另一项关键局限在于，诸如 GRPO [18] 这类广泛使用的基于组的优势函数方法，依赖于组内 rollout 之间的相对比较。当组内所有样本要么全对要么全错时，这些方法便无法提供有信息量的学习信号。
为解决这一问题，近期研究引入了特权信息来辅助策略 [27, 16, 19, 8]，这些信息通常以 oracle 前缀或中间解的形式呈现。这些方法通过使模型能够生成原本无法达到的成功轨迹，提升了解决难题的可行性。
然而，此类方法存在若干局限。首先，特权信息往往难以规模化，尤其是当其依赖于 oracle 解或昂贵的标注时 [27, 16]。其次，模型如何内化这些信息并在测试时无需其辅助仍能表现良好，其机制尚不明确 [8]。第三，许多方法假设了多轮或交互式场景 [19]，这可能与标准的单轮 RLVR 设定不符。
更重要的是，现有工作主要聚焦于提升在难题上生成正确轨迹的可行性。相比之下，我们的工作则针对一个互补性的挑战：即便成功轨迹已经可达，也要提升探索的多样性。
附录 B 策略引导的详细说明
策略生成提示词。
我们使用 gpt-4o-mini 为每个问题生成关键词级别的提示。在主要实验中，我们为每个问题生成两个提示；而在排名靠前的设置中，我们首先生成五个候选提示，然后基于 oracle 评估从中选择一个子集。
用于生成提示的确切提示词如下：
num_hints
num_hints
num_hints
提示
num_hints
策略引导提示词。
给定一个问题和一个可选的提示，我们构建提示词，鼓励模型遵循特定的推理策略。模型被指示使用预定义的分隔符，将其推理过程和最终答案明确分开。
reasoning_start
start_working_out
reasoning_end
end_working_out
solution_start
SOLUTION
solution_end
SOLUTION
system_prompt
reasoning_start
reasoning_end
然后
提供
你的
解决方案
在
solution_start
solution_end
def
build_messages
problem
str
system_prompt
context_block
if
hint
context_block
user_content
context_block
system_prompt
role
user
content
user_content
策略引导的效果。
为了评估策略引导的效果，我们在 DAPO-17k-Processed [24] 的 200 个问题上，对 Qwen3-4B-Instruct-2507 [21] 分别在有和没有策略引导的情况下采样了 8 次 rollout，并通过 LLM-as-a-judge 分析生成的 rollout 多样性。
LLM-as-a-judge 提示词。
为了分析生成 rollout 的多样性，我们采用 LLM-as-a-judge，使用 gpt-4o-mini 根据其底层推理策略对解决方案进行聚类，并统计不同解决方案模式的数量。给定一个问题及一组 rollout，模型被指示识别出概念上不同的解决方法的数量，同时忽略诸如措辞或微小计算差异等表面上的不同。
prompt
f
问题
n
problem_text
formatted_rollouts
附录 C 基线方法详情
GRPO 中的 Rollout 缩放。
在受控实验中，我们缩放每个提示的 rollout 数量，同时相应调整梯度累积步数和生成批次大小，如表 2 所示。这确保了在不同 rollout 设置下，整体优化动态保持可比性。
实现 POPE [16]。
为了将我们的方法与基于 oracle 前缀的方法进行比较，我们实现了自己的 POPE [16] 版本。我们遵循原论文，使用相同的提示词格式和数据集混合（即包含和不包含特权信息）。由于不同前期工作 [27, 16] 中 oracle 解决方案的长度各不相同，我们通过将 oracle 解决方案截断至其完整长度的一定比例作为前缀来对此进行标准化。
生成上下文示例。
我们提供了一个说明性示例，展示我们方法中使用的策略级上下文。这些上下文是轻量级的、关键词级别的提示，并不直接解决问题，而是引导模型进入不同的推理模式。重要的是，它们并非旨在提供中间步骤或解决方案，而是作为高层次的归纳偏置，以多样化探索过程。
Oracle 解决方案：
策略级上下文（我们的方法）：
附录 D 训练细节
框架。
我们使用 TRL [22] 来实现基线模型和我们的算法。
超参数。
| 参数 | 值 |
|---|---|
| LoRA 秩 | 32 |
| 最大提示词长度 | 2,048 |
| 最大补全长度 | 6,144 |
| RL 步数 | 500 |
| 批次大小 | 4 |
| 每个提示词的展开次数 | |
| 梯度累积步数 | |
| 生成批次大小 | |
| 温度 | 1 |
| 最小 | 0.0 |
| Top- | 0.95 |
| Top- | |
| 学习率 | |
| 学习率调度器 | 余弦 |
| 权重衰减 | 0.001 |
| 预热比例 | 0.05 |
| 优化器 | AdamW（8 位） |
| KL 系数 | 0 |
| Epsilon 低值 | 0.2 |
| Epsilon 高值 | 0.2（除非另有说明） |
| 1.1 | |
| 0.1 | |
| 0.5 | |
| 随机种子 | 42 |
我们训练中使用的超参数见表 2。
计算资源。
所有实验均使用 NVIDIA H200 140GB GPU。
附录 E 评估细节
在评估过程中，所有超参数与表 2 保持一致，但温度除外，其设置为 。
附录 F 案例研究细节
在本节中，我们提供来自图 4 所示案例研究的定性示例。
经 GRPO 训练的模型主要依赖坐标几何，并结合启发式对称性假设和逐案例面积分解。尽管这些方法偶尔能推进到部分解，但通常在得出最终答案之前，会导致过长的推导过程和截断的输出。
相比之下，NudgeRL 利用了鞋带公式策略，该策略可直接根据顶点坐标计算多边形面积。这一策略生成了更短且更可靠的推理轨迹，从而能够在生成预算内成功完成任务。
附录 G 完整评估结果
| AIME24 | AIME25 | AMC23 | MATH500 | 平均 | |
|---|---|---|---|---|---|
| 0.00 | 0.418 | 0.344 | 0.759 | 0.628 | 0.537 |
| 0.25 | 0.461 | 0.354 | 0.773 | 0.658 | 0.561 |
| 0.50 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| 0.75 | 0.458 | 0.361 | 0.796 | 0.649 | 0.566 |
| 采样方式 | AIME24 | AIME25 | AMC23 | MATH500 | 平均 |
|---|---|---|---|---|---|
| 随机 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| 排名最高 | 0.448 | 0.355 | 0.774 | 0.632 | 0.552 |
| AIME24 | AIME25 | AMC23 | MATH500 | 平均 | |
|---|---|---|---|---|---|
| 0.9 | 0.403 | 0.366 | 0.806 | 0.648 | 0.556 |
| 1.0 | 0.436 | 0.359 | 0.831 | 0.643 | 0.567 |
| 1.1 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| AIME24 | AIME25 | AMC23 | MATH500 | 平均 | |
|---|---|---|---|---|---|
| 0.0 | 0.423 | 0.362 | 0.826 | 0.628 | 0.560 |
| 0.1 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| 0.5 | 0.425 | 0.361 | 0.730 | 0.629 | 0.536 |
| 算法 | AIME24 | AIME25 | AMC23 | MATH500 | 平均 | |
|---|---|---|---|---|---|---|
| NudgeRL | 0.2 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| GRPO | 0.2 | 0.444 | 0.367 | 0.749 | 0.668 | 0.557 |
| 0.24 | 0.451 | 0.373 | 0.795 | 0.645 | 0.566 | |
| 0.28 | 0.443 | 0.372 | 0.793 | 0.648 | 0.564 | |
| 0.32 | 0.452 | 0.358 | 0.813 | 0.640 | 0.566 | |
| 0.36 | 0.432 | 0.338 | 0.845 | 0.647 | 0.565 | |
| 0.40 | 0.406 | 0.341 | 0.781 | 0.638 | 0.541 |
附录 H 更广泛的影响
本文提出了一种高效的框架，用于在具有可验证奖励的强化学习（RLVR）中进行结构化探索。从积极方面来看，我们的方法在不依赖极大 rollout 预算或昂贵 oracle 监督的情况下提高了探索效率，这有助于降低训练推理模型的计算成本，并提高小型研究团队的可及性。
然而，提升探索效率也可能推动日益强大的推理系统的发展，这些系统可能被以有害或非预期的方式滥用。因此，我们强调持续开展关于安全性、监督与负责任部署的研究的重要性。
Abstract
Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models. However, its effectiveness is fundamentally limited by exploration: the policy can only improve on trajectories it has already sampled. While increasing the number of rollouts alleviates this issue, such brute-force scaling is computationally expensive, and existing approaches that modify the optimization objective provide limited control over what is explored. In this work, we propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR. Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts to induce diverse reasoning trajectories without relying on expensive oracle supervision. To effectively learn from such structured exploration, we further propose a unified objective, which decomposes the reward signal into inter- and intra-context components and incorporates a distillation objective to transfer discovered behaviors back to the base policy. Empirically, NudgeRL outperforms standard GRPO with up to 8 larger rollout budgets, while outperforming oracle-guided RL baseline on average across five challenging math benchmarks. These results demonstrate that structured, context-driven exploration can serve as an efficient and scalable alternative to both brute-force rollout scaling and feasibility-oriented methods based on privileged information. Our code is available at https://github.com/tally0818/NudgeRL.
1 Introduction
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of large language models (LLMs) [20, 7]. By leveraging verifiable rewards, methods such as Group-Relative Policy Optimization (GRPO) [18] enable scalable post-training without requiring dense supervision. This paradigm has been successfully applied across a wide range of domains.
Despite its success, RLVR remains fundamentally limited by its ability to explore the space of reasoning trajectories. A natural approach is to scale the number of sampled rollouts, which increases the probability of discovering rare trajectories [5]. However, such brute-force scaling quickly becomes computationally prohibitive, motivating alternative approaches that improve exploration efficiency.
Recent work has sought to address this limitation by modifying the optimization objective, for example through entropy regularization or decoupled clipping [26, 24]. While these methods encourage broader exploration at the distribution level, they provide limited control over what is explored, and often fail to ensure coverage of semantically meaningful reasoning strategies. Another line of work leverages privileged information, such as oracle solutions or intermediate reasoning steps, to improve the feasibility of discovering correct trajectories [27, 16, 8, 19]. Although effective, these approaches are primarily feasibility-oriented and rely on strong supervision signals that are expensive to obtain and difficult to scale. Moreover, by guiding the policy toward a narrow set of predefined successful trajectories, they may limit exploration diversity and hinder the discovery of alternative reasoning strategies [25, 23].
In this work, we address the exploration bottleneck by explicitly structuring the reasoning space in a scalable manner. We propose NudgeRL, a framework that introduces Strategy Nudging during the exploration phase. Instead of relying on expensive oracle data, Strategy Nudging appends lightweight, heuristic text prompts (e.g., specific strategies for math problems or reasoning keywords) to the input. This deliberately forces the model to traverse distinct, diverse reasoning modes that it might otherwise ignore under purely naive sampling.
However, learning from such context-conditioned exploration introduces new challenges. Since rollouts are generated under different context-conditioned prompts, the samples are naturally partitioned into multiple distinct groups, where reward variation reflects both the intrinsic trajectory quality and context-specific biases, making standard group-wise advantage estimation unreliable. Furthermore, context forcing creates a mismatch between how trajectories are sampled and how the policy is finally used at inference time. Without intervention, improvements discovered under context-forced exploration may not transfer directly to the base policy. To address these challenges, we further introduce (i) an Inter-Intra group advantage to enable meaningful credit assignment across context-induced groups, and (ii) a distillation-augmented objective that explicitly transfers effective behaviors discovered during context-forced exploration back to the base policy.
Our approach enables structured and diversity-driven exploration while remaining fully compatible with standard RLVR pipelines. Empirically, NudgeRL achieves performance surpassing GRPO even when GRPO is given an larger rollout budget, while outperforming oracle-guided baselines. This demonstrates that scalable, diversity-oriented exploration can serve as an effective alternative to both brute-force rollout scaling and feasibility-driven privileged information.
2 Preliminaries
2.1 Group-Relative Policy Optimization (GRPO)
We consider an empirical distribution of prompts . For each prompt , a policy generates a group of rollouts , where each rollout is sampled as . Each rollout is evaluated by a verifiable reward function .
Unlike standard PPO [17], which typically estimates advantages using a learned value function, GRPO [18] derives advantages from group-wise rewards. For rollouts sampled from the same prompt , let denote the reward of rollout . The group-wise advantage is then defined as:
| (1) |
where and are the reward mean and standard deviation within the group, and is used for numerical stability. This yields a relative advantage estimate without training a value function.
The policy is then optimized with a PPO-style clipped objective:
| (2) |
Thus, GRPO retains PPO’s clipped objective while using group-relative advantages.
2.2 Motivation: From Exploration to Performance Gain
To understand why exploration is a fundamental bottleneck in RLVR, we look beyond trajectory-level rewards and examine how the probability mass of generated tokens shifts during training. Hu et al. [5] characterizes the expected one-step performance improvement () in RLVR as:
| (3) |
where and denote the total probability mass of correct and incorrect tokens, is the learning rate, and is the number of rollouts. and are the second moments of sampled correct and incorrect tokens, while and are those of unsampled correct and incorrect tokens. represents the net reward contribution from sampled tokens.
Since , the first two terms in Eq.˜3 are non-negative and drive learning forward. The third term, however, acts as a potential penalty. Because incorrect tokens typically dominate the probability mass (), a large , meaning the model has significant probability mass on correct trajectories that it simply fails to explore, creates a dominant negative force that hinders performance gain. Therefore, the core bottleneck of RLVR lies in the unexplored correct regions.
Limitations of rollout scaling.
To mitigate this penalty, a naive solution is to increase the rollout size . Hu et al. [5] shows that for a collection of tokens with probabilities , the expected unsampled second moment after draws is:
| (4) |
which decreases monotonically with . However, tokens with small decay slowly, so fully covering long-tail correct trajectories requires prohibitively large rollout budgets.
This highlights the limitation of blindly scaling to reduce the unexplored correct mass (). Long-tail correct trajectories remain unlikely to be sampled even under large , suggesting the need for a structured exploration mechanism that can efficiently expose such latent trajectories.
3 NudgeRL
We introduce NudgeRL, a framework for structured exploration and learning in RLVR. NudgeRL consists of three components: (i) Strategy Nudging, which conditions rollout generation on strategy-level contexts to induce diverse reasoning trajectories; and (ii) Inter-intra Group Advantage, a credit assignment method that enables controlled exploration and exploitation of strategies; and (iii) Distillation augmented RL objective to learn from context-conditioned rollouts and distill effective strategies into the policy under the original prompt for inference without external context.
3.1 Strategy Nudging: Structured Exploration via Strategy-Level Contexts
Given that prior work [5] alleviates the exploration bottleneck by reducing unsampled probability mass through larger rollout budgets, a natural question arises: how many rollouts are required to reliably discover a rare trajectory? To quantify this discovery cost, consider a rare trajectory with . The expected number of rollouts required to observe at least once is:
| (5) |
This implies that for low-probability trajectories, the required rollout budget grows prohibitively large. In practice, naive rollout scaling repeatedly samples from high-probability modes of the current policy, leading to diminishing returns in covering rare trajectories.
This motivates conditioning generation on a context that can shift the sampling distribution toward otherwise rare trajectories. If such a context increases the probability of a trajectory , i.e., , then its expected number of rollouts becomes:
| (6) |
Thus, contexts need not provide a solution; they can serve as lightweight controls that alter the sampling distribution and reduce the cost of discovering rare trajectories.
Strategy Nudging.
Even though context conditioning can improve exploration efficiency in principle, simply placing multiple contexts in a single prompt leaves the choice of strategy to the policy, which may ignore some contexts and repeatedly follow dominant reasoning patterns. To enforce coverage over contexts, we instead assign a single sampled context to each rollout before generation.
Let denote a pool of Strategy-level contexts for the original prompt . For each rollout index , we begin with sampling . To avoid relying exclusively on the context pool and to retain compatibility with the original prompt, we further apply context dropout. Specifically, we sample a mask and define the context as:
| (7) |
We then construct the final prompt , and generate . By varying across rollout indices, Strategy Nudging induces diversity at the input-conditioning level, rather than relying solely on sampling from a single prompt. Details on generating are in Appendix˜B.
Context-induced rollout diversity.
To verify that Strategy Nudging induces the intended diversity, we compare it against naive sampling without context conditioning. For each prompt, both methods generate 8 rollouts in total: Strategy Nudging samples 4 rollouts from each of 2 contexts without context dropout, whereas the baseline samples all 8 rollouts from the base policy under the original prompt. We then cluster the reasoning structures using an LLM-as-a-judge (gpt-4o-mini [15]) and measure the number of distinct clusters; additional details are provided in Appendix˜B.
As shown in Fig.˜1, Strategy Nudging more often increases the number of distinct reasoning structures relative to naive sampling, whereas the base policy frequently collapses to similar patterns. This suggests that Strategy Nudging diversifies exploration before any policy update is applied, allowing the rollout set to cover a broader range of reasoning modes under the same rollout budget.
3.2 Inter-Intra Group Advantage: Learning to Balance Exploration between Strategies
GRPO estimates advantages by comparing rewards among rollouts conditioned on the same prompt distribution. With Strategy Nudging, however, rollouts are drawn from context-conditioned prompts . A single group baseline therefore entangles reward variation induced by different contexts, distorting the relative advantage assigned to each rollout.
To address this, we propose the Inter-Intra Group Advantage, which assigns credit through two complementary signals: an intra-context signal, capturing trajectory quality under the same conditioning context, and an inter-context signal, capturing the relative reliability of the context itself.
Given sampled rollouts with rewards , we group them according to their assigned contexts. The set of context groups is defined as
| (8) |
For each group , we define the index set , which partitions all rollouts. We then compute both context-level and global reward baselines:
| (9) |
Using these baselines, we define the advantage as:
| (10) |
and are the mean and standard deviation of , and ensures numerical stability.
Because advantages determine direction of the policy update, they should remain consistent with the underlying rewards while allowing context-level preferences to affect credit assignment.
Proposition 3.1.
Consider two trajectories and sampled from context groups and , with rewards and , respectively. Let and denote the corresponding context means, and let and denote their advantages. In the binary reward setting, if , then:
| (11) |
Thus, for , a higher reward always receives a higher advantage, ensuring consistency with the underlying objective; context only affects the relative ordering among equal-reward trajectories. For equal-reward trajectories, controls the context-level preference: favors successes from lower-reward contexts, encouraging exploration of less typical contexts, whereas favors successes from higher-reward contexts, emphasizing more reliable contexts. The neutral case treats equal-reward trajectories identically across contexts; the case is illustrated in Fig.˜2 (a).
3.3 Training objective
Although Strategy Nudging improves exploration by sampling rollouts from context-conditioned prompts , the target policy at inference time should operate without external contexts. Therefore, useful trajectories discovered under must be transferred to the base policy .
To bridge this gap, we introduce an advantage-weighted distillation term following Song et al. [19], which directly updates the policy using trajectories sampled under the context-conditioned input :
| (12) |
Unlike standard behavior cloning, this formulation selectively emphasizes trajectories with high normalized advantage, ensuring that only useful behaviors discovered under diverse contexts contribute to the update of .
In parallel, we optimize the reinforcement learning objective on the context-conditioned policy:
| (13) |
The final objective combines both terms:
| (14) |
This objective induces a complementary learning dynamic. The RL term operates on the context-conditioned policy, improving exploration and reinforcing successful trajectories within each context. In contrast, the distillation term projects these improvements onto the base-prompt policy, enabling cross-context generalization. As a result, the model learns to reproduce effective reasoning strategies without relying on explicit context at inference time. Unlike GRPO in Eq.˜2, which samples and optimizes trajectories under the original prompt , NudgeRL performs RL on context-conditioned rollouts under while distilling high-advantage trajectories back into the base policy .
4 Experiments
| Model | Method | Rollouts () | AIME24 | AIME25 | AMC23 | MATH500 | APEX | Average |
|---|---|---|---|---|---|---|---|---|
| Qwen3-4B- Instruct | Base model | – | 0.374 | 0.352 | 0.653 | 0.592 | 0.036 | 0.402 |
| GRPO | 8 | 0.444 | 0.367 | 0.749 | 0.668 | 0.040 | 0.454 | |
| 16 | 0.454 | 0.355 | 0.840 | 0.655 | 0.045 | 0.470 | ||
| 32 | 0.451 | 0.370 | 0.881 | 0.674 | 0.058 | 0.487 | ||
| 64 | 0.415 | 0.324 | 0.848 | 0.641 | 0.027 | 0.451 | ||
| POPE†[16] | 8 | 0.460 | 0.337 | 0.838 | 0.652 | 0.048 | 0.467 | |
| \rowcolorciteblue!10 \cellcolorwhite | NudgeRL | 8 | 0.482 | 0.393 | 0.857 | 0.660 | 0.053 | 0.489 |
| Olmo3-7B- Instruct-SFT | Base model | – | 0.134 | 0.118 | 0.467 | 0.384 | 0.021 | 0.225 |
| GRPO | 8 | 0.187 | 0.159 | 0.537 | 0.434 | 0.025 | 0.268 | |
| 16 | 0.188 | 0.176 | 0.548 | 0.461 | 0.023 | 0.279 | ||
| 32 | 0.195 | 0.176 | 0.553 | 0.459 | 0.024 | 0.281 | ||
| 64 | 0.081 | 0.053 | 0.349 | 0.291 | 0.027 | 0.160 | ||
| POPE†[16] | 8 | 0.186 | 0.169 | 0.558 | 0.460 | 0.023 | 0.279 | |
| \rowcolorciteblue!10 \cellcolorwhite | NudgeRL | 8 | 0.190 | 0.179 | 0.563 | 0.468 | 0.025 | 0.285 |
4.1 Experimental Setup
Baselines.
We compare our method against (i) the base model without optimization, which serves as the reference point; (ii) GRPO with increasing rollout budgets, where , which evaluates naive rollout scaling as a brute-force exploration strategy; and (iii) POPE [16], which augments standard GRPO by appending prefixes of the oracle solution at the end of the base prompt, thereby alleviating the sparse reward signal bottleneck. Further details are provided in Appendix˜C.
Evaluation Datasets and Metrics.
AIME24 and AIME25, 30-problem olympiad-style high-school competitions [13]; AMC23, a 40-problem high-school contest benchmark [12]; the level-5 subset of MATH500, containing 134 difficult MATH problems [4]; and the Apex Shortlist, consisting of 48 advanced competition-style problems [1]. We report pass@1, estimated from 128 rollouts using the unbiased estimator of Chen et al. [2]. All solutions are automatically graded using math-verify [6]. Additional details are provided in Appendix˜E.
Implementation Details.
We apply NudgeRL to Qwen3-4B-Instruct-2507 [21] and Olmo-3-7B-Instruct-SFT [14] using DAPO-17k-Processed as a training set [24]. To construct the pool of contexts, we used gpt-4o-mini [15] to generate two strategy-level contexts per problem (e.g., Pythagorean theorem), and used them without additional verification (i.e., ). For the POPE baseline, oracle solutions were generated using DeepSeek Reasoner v3.2 [9]. We provide additional optimization details in Appendix˜D.
4.2 Main Results
NudgeRL matches larger-budget GRPO with fewer rollouts.
As shown in Tab.˜1, NudgeRL achieves the best average performance on both models while using only 8 rollouts per prompt. On Qwen3-4B-Instruct-2507, NudgeRL reaches 0.489 average pass@1, slightly outperforming the best GRPO result at 32 rollouts (0.487) and surpassing GRPO at 64 rollouts (0.451) with an 8 smaller rollout budget. On Olmo3-7B-Instruct-SFT, NudgeRL likewise improves over the best GRPO result, achieving 0.285 compared to 0.281 at 32 rollouts. These results indicate that larger rollout budgets alone are not sufficient: GRPO improves up to but degrades at on both models, suggesting instability under brute-force rollout scaling. In contrast, NudgeRL achieves stronger performance by improving the quality of exploration through Strategy Nudging, rather than relying on more sampled rollouts.
Comparison with oracle-prefix method.
We also compare with POPE [16], which augments GRPO by generating rollouts conditioned on the oracle solution prefixes. Unlike baselines relying on expensive, unscalable oracle hints [16] or text feedback [19], our approach ensures scalable diversity. We use a lightweight LLM (e.g., gpt-4o-mini) to cheaply generate unverified strategy-level contexts that induce multiple reasoning directions. Despite this weaker supervision, our method consistently outperforms oracle-guided baselines, demonstrating that structured exploration over diverse strategies is more effective than injecting narrow, privileged solution signals.
4.3 Efficient Coverage of Diverse Reasoning Modes
As discussed in Sec.˜3.1, relying solely on scaling the rollout budget suffers from severe sample inefficiency when discovering long-tail, low-probability reasoning modes. This is because naive rollout scaling repeatedly allocates computation to dominant trajectories. To empirically investigate how Strategy Nudging overcomes this exploration bottleneck and improves sample efficiency, we compare the training dynamics of NudgeRL against GRPO under progressively larger rollout budgets. We evaluate the model for every 50 training steps on the combined AIME24 and AIME25 benchmark by sampling 64 rollouts per problem and estimating pass@1 and pass@8.
As shown in Fig.˜3(b), NudgeRL improves faster than GRPO variants and remains the strongest method throughout most of training. By 200 steps, NudgeRL exceeds 0.42 on AIME24/25, while GRPO variants remain around or below 0.41 and show slower or less stable gains as the rollout budget increases. This suggests that Strategy Nudging improves sample efficiency by exposing useful reasoning trajectories earlier, rather than merely increasing sampled rollouts. Enlarging the number of samples () further validates this trend under the same training rollout budget. As shown in Fig.˜3(c), NudgeRL consistently outperforms GRPO-8 across the full range, which indicates that Strategy Nudging improves inference-time sample efficiency, requiring fewer generated solutions to reach the same level of .
4.4 Case Study
To examine the source of performance gains in NudgeRL, we analyze one AIME25 problem where the NudgeRL-trained model successfully sampled correct trajectories, while the GRPO-trained model entirely failed. We sampled 32 rollouts and categorized their dominant reasoning strategies.
As shown in Fig.˜4, both models predominantly relied on coordinate geometry. However, the GRPO-trained model additionally explored ineffective strategies such as symmetry assumptions and area decomposition, which consistently resulted in truncated solutions, causing all 32 trajectories to fail. While GRPO sampled the shoelace formula strategy only once, NudgeRL substantially increased its frequency and successfully exploited it to generate correct trajectories.
This behavior highlights the complementary roles of our framework: Strategy Nudging exposes rare but effective reasoning modes such as the shoelace-formula strategy, while the Inter-Intra Group Advantage reinforces and exploits such reliable strategies once discovered. Details are in Appendix˜F.
4.5 Effect of Contexts during training
We also report the dropout reward mean () and the hinted reward mean () during training of Qwen3-4B-Instruct-2507 with NudgeRL. As shown in Fig.˜5, both rewards improve together throughout training, suggesting that trajectories discovered under context-conditioned exploration are successfully transferred to the base policy through the distillation objective. Interestingly, the dropout reward occasionally exceeds the hinted reward during training. This contrasts with prior feasibility-oriented methods based on privileged information [16, 27, 8, 19]. In ours, primary role of context is not to directly simplify the problem, but to induce diverse reasoning trajectories that can later be internalized by the context-free policy.
4.6 Underlying Mechanism of NudgeRL
To further understand the source of performance gains in NudgeRL, we conduct a series of controlled experiments using Qwen3-4B-Instruct-2507 [21] on a subset of benchmarks.
Ablation.
As shown in Fig.˜7(a), a moderate dropout rate () consistently yields the best performance across benchmarks. Context dropout plays a dual role: it enables exploration beyond fixed contexts by occasionally reverting to the base prompt, while also stabilizing group-wise statistics through a more balanced sample distribution. When , exploration is restricted to predefined contexts, whereas large values diminish the influence of context forcing. These results suggest that maintaining a balanced mixture of context-conditioned and context-free samples is important for achieving both diverse exploration and stable optimization.
Hint Sampling.
We study how the quality of sampled contexts affects performance by comparing two strategies: random sampling and top-ranked selection. In the top-ranked setting, we first generate a pool of five candidate contexts, and then select the two that yield the largest improvement in for each problem, as measured by oracle evaluation.
As shown in Fig.˜7(b), random sampling consistently outperforms top-ranked selection in terms of . While top-ranked contexts ensure more correctness, they tend to concentrate on a narrow set of reasoning strategies. In contrast, random sampling induces a broader distribution over plausible trajectories, resulting in more effective exploration under limited rollout budgets.
These results suggest that, within our framework, the primary role of context is not to provide the single best hint, but to promote diversity in reasoning. Consequently, simple random sampling is not only sufficient, but also preferable for scalable and effective context-based exploration.
Exploration-Exploitation trade-off via .
Fig.˜7(a) presents the effect of varying , where achieves the best performance. This trend aligns with our Proposition 3.1 in Sec.˜3.2. Since strategy nudging already ensures sufficient diversity at the sampling stage, increasing does not hinder exploration across contexts. Instead, it strengthens exploitation within each problem by prioritizing trajectories from more reliable contexts. This leads to more consistent learning of high-quality solutions per instance, explaining the observed performance gains at .
Distillation Coefficient.
As shown in Fig.˜7(b), removing the distillation term () results in a clear performance drop, indicating that explicitly transferring context-discovered trajectories to the base policy is essential. However, overly large values also degrade performance, likely due to over-constraining the policy toward sampled trajectories. A moderate coefficient () achieves the best results, suggesting that distillation should complement the underlying RL objective.
4.7 Comparison with scaling.
We further compare our algorithm with decoupled clipping [24]: where controls the strength of policy updates by amplifying the contribution of successful trajectories. Increasing therefore allows more aggressive policy updates toward positive-advantage trajectories. As shown in Fig.˜7(c), increasing generally improves GRPO performance in the moderate regime used in prior works [18, 24]. However, our method with consistently outperforms GRPO across the entire scaling range from moderate to extreme values. This suggests that improving exploration quality is more effective than simply increasing the magnitude of stochastic policy updates. Additionally, under the more extreme scaling adopted in recent RLVR settings [10], GRPO sharply deteriorates at . We argue that this degradation highlights a limitation of purely stochastic distribution-level exploration: increasing update magnitude alone provides little control over what is explored.
The complete results of the evaluation are given in the Appendix˜G.
5 Conclusion
In this work, we introduced NudgeRL, a framework for structured exploration in RLVR. Our approach leverages Strategy Nudging to induce diverse reasoning trajectories by sampling from lightweight, strategy-level context-conditioned distributions, and learns from them via distillation augmented RL objective. Empirically, NudgeRL achieves superior performance compared to GRPO using up to 8 larger rollout budgets, and further outperforms oracle prefix-based baselines across models.
Limitations & Future Work
A practical consideration of NudgeRL is the cost of generating strategy-level contexts. However, this is an offline process performed once prior to training, using a lightweight LLM (e.g., gpt-4o-mini), and the resulting contexts can be reused across training runs without additional overhead. A more fundamental limitation lies in how contexts are generated independently of the model being trained. The benefit of Context Forcing stems from inducing trajectories that are unlikely under the current policy. As training progresses, however, a fixed context pool may become less informative as the policy adapts. A promising direction for future work is model-adaptive context generation, which dynamically constructs contexts tailored to the current policy’s blind spots, potentially yielding more consistent exploration gains throughout training.
References
- [1] M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev (2025-02) MathArena: evaluating llms on uncontaminated math competitions. SRI Lab, ETH Zurich. External Links: Link Cited by: §4.1.
- [2] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §4.1.
- [3] J. Deng, J. Chen, Z. Chen, W. X. Zhao, and J. Wen (2025) Decomposing the entropy-performance exchange: the missing keys to unlocking effective reinforcement learning. arXiv preprint arXiv:2508.02260. Cited by: §A.2.
- [4] D. Hendrycks, C. Burns, S. Basart, A. Zou, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. NeurIPS. Cited by: §4.1.
- [5] J. Hu, M. Liu, X. Lu, F. Wu, Z. Harchaoui, S. Diao, Y. Choi, P. Molchanov, J. Yang, J. Kautz, et al. (2025) Brorl: scaling reinforcement learning via broadened exploration. arXiv preprint arXiv:2510.01180. Cited by: §A.2, §1, §2.2, §2.2, §3.1.
- [6] HuggingFace (2024) Math-verify: a toolkit for verifying mathematical reasoning. Note: https://github.com/huggingface/Math-VerifyAccessed 2026-05-06 Cited by: §4.1.
- [7] N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. (2024) Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §1.
- [8] B. Liao, H. Dong, X. Xu, C. Monz, and J. Bian (2026) Self-hinting language models enhance reinforcement learning. arXiv preprint arXiv:2602.03143. Cited by: §A.3, §A.3, §1, §4.5.
- [9] A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, et al. (2025) Deepseek-v3. 2: pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Cited by: §A.1, §A.1, §4.1.
- [10] M. Liu, S. Diao, X. Lu, J. Hu, X. Dong, Y. Choi, J. Kautz, and Y. Dong (2025) Prorl: prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864. Cited by: §A.1, §4.7.
- [11] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: A critical perspective. arXiv 2503.20783. External Links: Link Cited by: §A.1.
- [12] Mathematical Association of America (2023) American mathematics competitions. Note: https://www.maa.org/math-competitions Cited by: §4.1.
- [13] Mathematical Association of America (2025) AIME: american invitational mathematics examination. Note: https://www.maa.org/math-competitions Cited by: §4.1.
- [14] T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi (2025) Olmo 3. External Links: 2512.13961, Link Cited by: §4.1.
- [15] OpenAI (2024) GPT-4o mini. Note: https://openai.com/ko-KR/index/gpt-4o-mini-advancing-cost-efficient-intelligence/Accessed: 2026-05-04 Cited by: §3.1, §4.1.
- [16] Y. Qu, A. Setlur, V. Smith, R. Salakhutdinov, and A. Kumar (2026) POPE: learning to reason on hard problems via privileged on-policy exploration. arXiv preprint arXiv:2601.18779. Cited by: §A.3, §A.3, Appendix C, Appendix C, §1, §4.1, §4.2, §4.5, Table 1, Table 1.
- [17] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.1.
- [18] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv 2402.03300. External Links: Link Cited by: §A.1, §A.1, §A.3, §1, §2.1, §4.7.
- [19] Y. Song, L. Chen, F. Tajwar, R. Munos, D. Pathak, J. A. Bagnell, A. Singh, and A. Zanette (2026) Expanding the capabilities of reinforcement learning via text feedback. arXiv preprint arXiv:2602.02482. Cited by: §A.3, §A.3, §1, §3.3, §4.2, §4.5.
- [20] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025) Kimi k1. 5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: §A.1, §1.
- [21] Q. Team (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: Appendix B, §4.1, §4.6.
- [22] TRL: Transformers Reinforcement Learning External Links: Link Cited by: Appendix D.
- [23] F. Wu, W. Xuan, X. Lu, Z. Harchaoui, and Y. Choi (2025) The invisible leash: why RLVR may not escape its origin. arXiv 2507.14843. External Links: Link Cited by: §1.
- [24] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source LLM reinforcement learning system at scale. arXiv 2503.14476. External Links: Link Cited by: §A.1, §A.1, §A.2, Appendix B, §1, §4.1, §4.7.
- [25] Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang (2025) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. arXiv 2504.13837. External Links: Link Cited by: §1.
- [26] X. Zhang, X. Yuan, D. Huang, W. You, C. Hu, J. Ruan, K. Chen, and X. Hu (2025) Rediscovering entropy regularization: adaptive coefficient unlocks its potential for llm reinforcement learning. arXiv preprint arXiv:2510.10959. Cited by: §A.2, §1.
- [27] X. Zhang, Z. Huang, Y. Li, C. Ni, J. Chen, and S. Oymak (2025) Bread: branched rollouts from expert anchors bridge sft & rl for reasoning. arXiv preprint arXiv:2506.17211. Cited by: §A.3, §A.3, Appendix C, §1, §4.5.
Appendix A Related Work
A.1 Reinforcement Learning with Verifiable Rewards
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning abilities of large language models[20, 18, 24, 9]. By leveraging automatically verifiable signals, such as exact answers in mathematics or test-case correctness in code generation, RLVR enables effective policy optimization without dense human supervision.
A representative approach is Group-Relative Policy Optimization (GRPO) [18], which replaces value function estimation with group-wise comparisons among sampled rollouts, deriving advantages from relative reward differences within each group. Building on this formulation, subsequent work has introduced improvements such as decoupled clipping [24] and alternative normalization strategies [11] to enhance training stability.
These methods have been successfully applied across a range of reasoning tasks and model scales [10, 9], establishing RLVR as a standard post-training approach for LLMs. However, their effectiveness fundamentally depends on exploration: the policy can only improve on trajectories it has already sampled. As a result, insufficient exploration directly limits learning, making it a key bottleneck in RLVR. We next examine how prior work addresses this challenge.
A.2 Exploration in RLVR
A straightforward approach to improving exploration is to scale the number of sampled rollouts. Prior work has shown that such rollout scaling can significantly improve performance by reducing the probability mass of un-sampled region[5]. However, this approach is computationally expensive and often impractical at scale.
More commonly, recent methods attempt to encourage exploration through objective design, such as entropy regularization [26, 3] or decoupled clipping [24]. While these approaches can steer the update toward exploration, they do not guarantee that useful or rare modes are actually sampled during training. In other words, shaping the distribution does not necessarily ensure coverage of meaningful trajectories, leaving exploration fundamentally limited.
Moreover, such distribution-level exploration is inherently stochastic and unconstrained, which can perturb the policy in semantically undesirable directions. Increasing entropy or aggressively reweighting probabilities may encourage the model to explore low-probability regions, but without any structural guidance, this often leads to incoherent or unproductive trajectories rather than meaningful reasoning strategies. As a result, these approaches lack control over how the policy explores, and fail to provide structured, strategy-level exploration that targets diverse and semantically valid modes of reasoning.
A.3 Usage of Privileged Information
Another key limitation of widely used group-based advantage methods, such as GRPO [18], is that they rely on relative comparisons within a group of rollouts. When all samples in a group are either correct or incorrect, these methods fail to provide informative learning signals.
To address this issue, recent works have introduced privileged information to assist the policy [27, 16, 19, 8], often in the form of oracle prefixes or intermediate solutions. These approaches improve the feasibility of solving hard problems by enabling the model to generate successful trajectories that would otherwise be unreachable.
However, such methods come with several limitations. First, privileged information is often difficult to scale, especially when it relies on oracle solutions or expensive annotations [27, 16]. Second, the mechanism by which the model internalizes this information and performs well without it at test time remains unclear [8]. Third, many approaches assume multi-turn or interactive settings [19], which may not align with standard single-turn RLVR setups.
More importantly, existing work primarily focuses on improving the feasibility of generating correct trajectories on difficult problems. In contrast, our work targets a complementary challenge: improving the diversity of exploration, even when successful trajectories are already attainable.
Appendix B Details on Strategy Nudging
Strategy Generating Prompt.
We use gpt-4o-mini to generate keyword-level hints for each problem. For the main experiments, we generate two hints per problem, while in the top-ranked setting, we first generate five candidate hints and select a subset based on oracle evaluation.
The exact prompt used for hint generation is as follows:
num_hints
num_hints
num_hints
Hint
num_hints
Strategy Nudging prompt.
Given a problem and an optional hint, we construct prompts that encourage the model to follow a specific reasoning strategy. The model is instructed to explicitly separate its reasoning process and final answer using predefined delimiters.
reasoning_start
start_working_out
reasoning_end
end_working_out
solution_start
SOLUTION
solution_end
SOLUTION
system_prompt
reasoning_start
reasoning_end
Then
provide
your
solution
between
solution_start
solution_end
def
build_messages
problem
str
system_prompt
context_block
if
hint
context_block
user_content
context_block
system_prompt
role
user
content
user_content
Effect of Strategy Nudging.
To evaluate the effect of Strategy Nudging, we sample 8 rollouts from Qwen3-4B-Instruct-2507 [21] on 200 problems from DAPO-17k-Processed [24], both with and without Strategy Nudging, and analyze the resulting rollout diversity via LLM-as-a-judge.
LLM-as-a-judge prompt.
To analyze the diversity of generated rollouts, we employ an LLM-as-a-judge using gpt-4o-mini to cluster solutions based on their underlying reasoning strategies and count the number of distinct solution modes. Given a problem and a set of rollouts, the model is instructed to identify the number of conceptually distinct solution approaches, while ignoring superficial differences such as phrasing or minor computational variations.
prompt
f
Problem
n
problem_text
formatted_rollouts
Appendix C Details on Baselines
Rollout Scaling in GRPO.
For controlled experiments, we scale the number of rollouts per prompt while adjusting the gradient accumulation steps and generation batch size accordingly, as summarized in Tab.˜2. This ensures that the total optimization dynamics remain comparable across different rollout settings.
Implementing POPE [16].
To compare our method with oracle prefix-based approaches, we implement our own version of POPE [16]. We follow the original paper in using the same prompt format and dataset mixture (i.e., with and without privileged information). Since the length of oracle solutions varies across prior works [27, 16], we standardize this by truncating the oracle solution to of its full length when used as a prefix.
Example of Generated Contexts.
We provide an illustrative example of the strategy-level contexts used in our method. These contexts are lightweight, keyword-level hints that do not directly solve the problem, but instead steer the model toward distinct reasoning modes. Importantly, they are not intended to provide intermediate steps or solutions, but rather to act as high-level inductive biases that diversify exploration.
Oracle solution:
Strategy-level contexts(ours):
Appendix D Training Detail
Framework.
We used TRL [22] for implementing baselines and our algorithm.
Hyperparameters.
| Parameter | Value |
|---|---|
| LoRA rank | 32 |
| Max prompt length | 2,048 |
| Max completion length | 6,144 |
| RL steps | 500 |
| Batch size | 4 |
| Rollouts per prompt | |
| Gradient accumulation steps | |
| Generation batch size | |
| Temperature | 1 |
| Min- | 0.0 |
| Top- | 0.95 |
| Top- | |
| Learning rate | |
| LR scheduler | cosine |
| Weight decay | 0.001 |
| Warmup ratio | 0.05 |
| Optimizer | AdamW (8-bit) |
| KL coefficient | 0 |
| Epsilon low | 0.2 |
| Epsilon high | 0.2 (unless specified) |
| 1.1 | |
| 0.1 | |
| 0.5 | |
| Random seed | 42 |
The hyperparameters we used in training are given in Tab.˜2.
Compute resources.
For all experiments, we used NVIDIA H200 140GB GPUs.
Appendix E Details on Evaluation
During evaluation, all hyperparameters are kept identical to Tab.˜2, except for the temperature, which is set to .
Appendix F Details on Case study
In this section, we provide qualitative examples from the case study presented in Fig.˜4.
The GRPO-trained model predominantly relied on coordinate geometry combined with heuristic symmetry assumptions and case-by-case area decomposition. Although these approaches occasionally progressed toward partial solutions, they frequently resulted in excessively long derivations and truncated outputs before reaching the final answer.
In contrast, NudgeRL exploited the shoelace-formula strategy, which directly computes polygon areas from vertex coordinates. This strategy produced substantially shorter and more reliable reasoning trajectories, enabling successful completion within the generation budget.
Appendix G Full Evaluation Results
| AIME24 | AIME25 | AMC23 | MATH500 | Average | |
|---|---|---|---|---|---|
| 0.00 | 0.418 | 0.344 | 0.759 | 0.628 | 0.537 |
| 0.25 | 0.461 | 0.354 | 0.773 | 0.658 | 0.561 |
| 0.50 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| 0.75 | 0.458 | 0.361 | 0.796 | 0.649 | 0.566 |
| Sampling | AIME24 | AIME25 | AMC23 | MATH500 | Average |
|---|---|---|---|---|---|
| Random | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| Top ranked | 0.448 | 0.355 | 0.774 | 0.632 | 0.552 |
| AIME24 | AIME25 | AMC23 | MATH500 | Average | |
|---|---|---|---|---|---|
| 0.9 | 0.403 | 0.366 | 0.806 | 0.648 | 0.556 |
| 1.0 | 0.436 | 0.359 | 0.831 | 0.643 | 0.567 |
| 1.1 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| AIME24 | AIME25 | AMC23 | MATH500 | Average | |
|---|---|---|---|---|---|
| 0.0 | 0.423 | 0.362 | 0.826 | 0.628 | 0.560 |
| 0.1 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| 0.5 | 0.425 | 0.361 | 0.730 | 0.629 | 0.536 |
| Algorithm | AIME24 | AIME25 | AMC23 | MATH500 | Average | |
|---|---|---|---|---|---|---|
| NudgeRL | 0.2 | 0.482 | 0.393 | 0.857 | 0.660 | 0.598 |
| GRPO | 0.2 | 0.444 | 0.367 | 0.749 | 0.668 | 0.557 |
| 0.24 | 0.451 | 0.373 | 0.795 | 0.645 | 0.566 | |
| 0.28 | 0.443 | 0.372 | 0.793 | 0.648 | 0.564 | |
| 0.32 | 0.452 | 0.358 | 0.813 | 0.640 | 0.566 | |
| 0.36 | 0.432 | 0.338 | 0.845 | 0.647 | 0.565 | |
| 0.40 | 0.406 | 0.341 | 0.781 | 0.638 | 0.541 |
Appendix H Broader Impacts
This paper proposes an efficient framework for structured exploration in reinforcement learning with verifiable rewards (RLVR). On the positive side, our method improves exploration efficiency without relying on extremely large rollout budgets or expensive oracle supervision, which may help reduce the computational cost of training reasoning models and improve accessibility for smaller research groups.
However, improving exploration efficiency may also contribute to the development of increasingly capable reasoning systems, which could be misused in harmful or unintended ways. We therefore emphasize the importance of continued research on safety, oversight, and responsible deployment.