# 论在线策略蒸馏的几何特性

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-05 08:00
- AIHOT 分数：58
- AIHOT 链接：https://aihot.virxact.com/items/cmq61s9bh04ctsl5ik49gx7j1
- 原文链接：https://arxiv.org/abs/2606.07082

## AI 摘要

研究对比了在线策略蒸馏（OPD）与监督微调（SFT）及带可验证奖励的强化学习（RLVR）在参数空间中的更新轨迹。OPD的更新影响更少权重，更强地避开主方向，且约束比RLVR松弛。OPD表现出子空间锁定：累积更新快速进入低维通道，且锁定子空间对OPD功能足够。控制实验表明，稀疏化更新token或off-policy生成不改变秩动态，而混合RLVR目标会改变。结论：OPD并非SFT与RLVR的中间点，而具有自身独特的更新几何。

## 正文

On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.