超越模仿:R2-OPD 按推理进展过滤在线策略蒸馏

HuggingFace Daily Papers(社区热门论文)·2026-08-19 08:00·7天前
AI 导读

在线策略蒸馏(OPD)假设教师奖励能恰当反映推理进展,但实际中二者常冲突,推理进步明显的步骤可能因偏离教师输出而获较低奖励。为此研究者提出 R2-OPD,构建教师奖励与独立估计进展奖励的轨迹内双排序,在排序不一致时抑制蒸馏奖励,减少与推理进展冲突的监督。该方法在推理性能上较标准 OPD 持续提升。

HuggingFace Daily Papers(社区热门论文)
41AI 编辑部评分,满分 100

超越模仿:R2-OPD 按推理进展过滤在线策略蒸馏

2026-08-19 08:00· 7天前
AI 导读

在线策略蒸馏(OPD)假设教师奖励能恰当反映推理进展,但实际中二者常冲突,推理进步明显的步骤可能因偏离教师输出而获较低奖励。为此研究者提出 R2-OPD,构建教师奖励与独立估计进展奖励的轨迹内双排序,在排序不一致时抑制蒸馏奖励,减少与推理进展冲突的监督。该方法在推理性能上较标准 OPD 持续提升。

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org