# β-OPSD：用策略优化推导、用自蒸馏训练

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-07-30 08:00
- AIHOT 分数：43
- AIHOT 链接：https://aihot.virxact.com/items/cms92atvy01o4ro9kqa2u06iz
- 原文链接：https://arxiv.org/abs/2607.28582

## AI 摘要

β-OPSD将vanilla OPSD重新定义为策略优化家族中β=1的特例，把KL惩罚权重β变成可控正则化参数，其最优策略是参考策略与特权教师之间的几何插值。该方法将闭式解转为蒸馏目标，通过混合token级logits高效实现，避免直接强化学习的高成本和高方差。在数学推理基准上，β-OPSD一致优于vanilla OPSD，提升优化稳定性和推理性能。

## 正文

On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the $β=1$ member of a broader policy-optimization family, where $β$ weights the KL penalty anchoring the student to a reference policy. This equivalence turns $β$ from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce $β$-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of $β$ selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that $β$-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.