Trust Region On-Policy蒸馏方法（TrOPD）
阅读原文· arxiv.org针对On-Policy蒸馏（OPD）在师生模型分布差异较大时训练不稳定的问题，本文提出置信区间On-Policy蒸馏（TrOPD）方法。该方法核心是采用置信区间On-Policy学习，仅在教师提供可靠监督的区域进行蒸馏；结合异常值估计策略处理不可靠区域；并通过离线引导鼓励向可靠区域探索。实验表明，TrOPD在数学推理、代码生成及通用基准测试上均优于包括OPD、EOPD和REOPD在内的现有OPD基线。
On-Policy Distillation (OPD) is a fundamental technique for efficient post-training of large language models (LLMs), with broad applications in agent learning, multi-task enhancement, and model compression. However, OPD training becomes unstable when the teacher and student distributions differ substantially, as teacher supervision on student-generated tokens may yield unreliable policy gradients and even cause optimization failure. This work addresses reliable on-policy token-level supervision through credit assignment strategies, and proposes Trust Region On-Policy Distillation, TrOPD. It features the following characteristics: 1) Trust-Region On-Policy Learning: TrOPD performs OPD only in regions where the teacher provides reliable supervision, mitigating the optimization difficulty of the K1 reverse-KL estimator under distribution mismatch. 2) Outlier Estimation: For outlier regions, we explore gradient clipping, masking, and forward-KL estimation to reduce the adverse effects of unreliable supervision. 3) Off-Policy Guidance: The student continues generation from teacher prefixes and uses forward KL to imitate off-policy guidance, encouraging on-policy exploration toward reliable regions. Experiments show that TrOPD consistently outperforms SoTA OPD baselines, including OPD, EOPD, and REOPOLD, across mathematical reasoning, code generation, and general-domain benchmarks.