# 进展优势：后训练中被忽视的免费午餐--面向LLM智能体的步骤级评分信号

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-24 08:00
- AIHOT 分数：64
- AIHOT 链接：https://aihot.virxact.com/items/cmqwqyy1i006fslikpe1wa3v3
- 原文链接：https://arxiv.org/abs/2606.26080

## AI 摘要

研究表明，强化学习后训练本身即可提供有效的步骤级评分信号，无需单独训练奖励模型。研究者在随机马尔可夫决策过程中推导出隐式优势函数——进展优势，即RL训练后策略与参考策略的对数概率比恰好还原最优优势函数。该信号无需人工标注、领域无关，且是标准RL后训练管线的副产品。在五个基准和四个模型族上，进展优势在测试时缩放、不确定性量化和失败归因三项应用中持续优于基于置信度的基线，甚至超越专门训练的奖励模型。

## 正文

Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.