# 揭密在线策略蒸馏：何时有益、何时有害及原因

- 来源：Apple Machine Learning Research（RSS）
- 发布时间：2026-07-09 08:00
- AIHOT 分数：52
- AIHOT 链接：https://aihot.virxact.com/items/cmre05t8p000fihwk60qqr36x
- 原文链接：https://machinelearning.apple.com/research/unmasking-on-policy-distillation

## AI 摘要

Apple机器学习研究团队提出训练无关诊断框架，以每个token、每个问题、每个教师的分辨率分析on-policy蒸馏。通过可扩展targeted-rollout算法估计理想梯度，并计算蒸馏梯度与理想梯度的余弦相似度（梯度对齐分数）。实验发现，蒸馏指导在错误rollouts上的对齐程度显著高于正确rollouts；最优蒸馏上下文取决于学生模型容量和目标任务，无通用配置。这些发现推动每任务、每token的诊断分析。

## 正文

research area Methods and Algorithms, research area Speech and Natural Language Processing

content type paperpublished July 2026

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

AuthorsMohammadreza Armandpour*, Fatih Ilhan*, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar

View publication

On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under which it is detrimental. Which teacher model should be used, and in the case of self-distillation, which specific context should serve as the supervisory signal? Does the optimal choice vary from one token to the next? At present, addressing these questions typically requires costly training runs whose aggregate performance metrics obscure the dynamics at the level of individual tokens. We introduce a training-free diagnostic framework that operates at the highest resolution: per token, per question, and per teacher. We derive an ideal per-node gradient defined as the parameter update that maximally increases the student’s probability of success. We then develop a scalable targeted-rollout algorithm to estimate this gradient efficiently, even for long chains of intermediate thoughts. The gradient alignment score, defined as the cosine similarity between this ideal gradient and any given distillation gradient, quantifies the extent to which a particular configuration approximates the ideal signal. Across a range of self-distillation settings and external teacher models, we observe that distillation guidance exhibits substantially higher alignment with the ideal on incorrect rollouts than on correct ones, where the student already performs well and the teacher’s signal tends to become noisy. Furthermore, we find that the optimal distillation context depends jointly on the student model’s capacity and the target task, and that no single universally effective configuration emerges. These findings motivate the use of per-task, per-token diagnostic analyses for distillation.

* Equal contribution

Related readings and updates.

Distillation Scaling Laws

July 1, 2025research area Methods and Algorithms, research area Speech and Natural Language Processingconference ICML

We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a…

Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution

June 10, 2021research area Computer VisionWorkshop at CVPR

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output…

Discover opportunities in Machine Learning.

Our research in machine learning breaks new ground every day.

Work with us