# GRAIL：面向可验证奖励强化学习的梯度重加权优势方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-03 08:00
- AIHOT 分数：63
- AIHOT 链接：https://aihot.virxact.com/items/cmpytyvum03assli3vy6zxxgh
- 原文链接：https://arxiv.org/abs/2606.04889

## AI 摘要

可验证奖励强化学习（如GRPO）常用统一的序列级优势更新所有token，稀释了梯度信号。GRAIL提出内在的逐token优势重加权方法，利用梯度激活显著度为对最终答案更敏感的token赋予更高权重。在Qwen3、R1-distilled和OctoThinker家族共5个模型上的评估显示，GRAIL一致优于GRPO，平均准确率提升3.60%，Pass@3提升3.05%，无需过程级监督即可实现细粒度推理对齐。

## 正文

Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical reasoning in Large Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained reasoning alignment can be achieved without process-level supervision.