# STARE：基于惊喜度的Token级优势重加权实现策略熵稳定

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-17 08:00
- AIHOT 分数：57
- AIHOT 链接：https://aihot.virxact.com/items/cmqjhshah03gbslmhwa1g87d1
- 原文链接：https://arxiv.org/abs/2606.19236

## AI 摘要

STARE是一种基于惊喜度的token级优势重加权方法，用于解决GRPO等可验证奖励强化学习中常见的策略熵崩溃问题。通过对GRPO下token级熵动态的一阶梯度分析，STARE识别出熵关键token子集并选择性重加权其有效优势，同时引入目标熵闭环门控实现稳定熵调节。在1.5B至32B参数的模型上，覆盖短思维链、长思维链和多轮工具使用三类任务，STARE能稳定训练数千步，保持策略熵在目标带内。在AIME24和AIME25基准上，STARE平均准确率高出DAPO等基线4%–8%，且反思token和响应长度协同增长。代码已开源。

## 正文

Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.