# 基于人类反馈的强化学习的另一面：奖励模型自监督改进的策略内反馈

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-29 08:00
- AIHOT 分数：63
- AIHOT 链接：https://aihot.virxact.com/items/cmpus0v0i00fzsl0zv65jzm7f
- 原文链接：https://arxiv.org/abs/2605.30888

## AI 摘要

构建用于大语言模型对齐的强奖励模型，其瓶颈在于获取多样化、可靠的人类偏好数据成本高昂，且当策略模型超出静态奖励模型训练范围时问题加剧。为此，研究提出SAVE框架，它利用价值函数对策略内生成的响应进行评分，将该评分作为反馈信号用于奖励模型的自监督训练。该框架通过提示特定的价值头作为自适应锚点，将评分后的响应转化为监督信号，计算奖励模型优势值并过滤模糊样本，最终通过对比目标更新模型。在六个多样化基准测试上的严格评估验证了其有效性，结果在所有数据集上超越基线，并在GRPO、RLOO、GSPO三种RL算法及不同策略主干上保持一致改进。

## 正文

Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.