ReNIO:为大语言模型同策略蒸馏重加权负轨迹重要性
阅读原文· arxiv.org同策略蒸馏平等对待所有学生生成输出。实验发现,仅用错误输出训练效果优于仅用正确输出,因错误输出保留模型能力边界附近的探索性推理。为此,ReNIO利用学生-教师概率比识别导致错误推理轨迹的关键token,聚合为归一化样本权重,无需观测最终答案正确性即可自动赋予负轨迹更高权重。在数学推理和代码生成任务上,ReNIO同时提升同策略蒸馏和同策略自蒸馏,对Qwen3-1.7B最高提升8.90%,对R1-Distill-Qwen-7B最高提升10.00%。代码已开源。
On-policy distillation (OPD) improves LLM reasoning by training a student model on its own generated outputs, but standard OPD treats all student-generated outputs (SGOs) equally regardless of their informativeness. We observe a consistent asymmetry in controlled filtering experiments: in both OPD and on-policy self distillation (OPSD), training only on incorrect SGOs outperforms training only on correct ones. Our further analysis suggests that models trained on correct-only SGOs tend to generate shorter reasoning traces and show weaker reflection behavior, while incorrect SGOs better preserve exploratory reasoning near the model's capability boundary. To exploit this signal without requiring full answer-containing rollouts, we introduce ReNIO, which Reweights Negative trajectory Importance for LLM On-policy distillation. By using the student-to-teacher probability ratio, ReNIO identifies pivotal tokens leading to wrong reasoning traces and aggregates their information into a normalized sample weight, inherently assigning larger weights to likely negative trajectories without observing the correctness of final-answer. Since Re-NIO only uses prefix-conditioned token probabilities, it preserves OPD's prefix training advantage over full-rollout reinforcement learning. Across both mathematical reasoning and code generation tasks, ReNIO improves both OPD and OPSD, with representative relative gains of up to 8.90% for Qwen3-1.7B and 10.00% for R1-Distill-Qwen-7B on mathematical reasoning benchmarks. Code repo: https://github.com/BDML-lab/ReNIO.