# 知识蒸馏在AI后训练中影响力正在下降

- 来源：Nathan Lambert (@natolambert)
- 发布时间：2026-07-22 01:15
- AIHOT 分数：52
- AIHOT 链接：https://aihot.virxact.com/items/cmrux92cy005gbinvy9e535zk
- 原文链接：https://x.com/natolambert/status/2079616308203942332

## AI 摘要

知识蒸馏在AI后训练中的关键作用正从RL阶段转向SFT/中期训练阶段，用于播种推理行为。SFT蒸馏数据通常通过API获取推理token生成，规模约100万条高质量提示-完成对，但蒸馏对最终模型性能的贡献比例正在缩小。随着RL规模化及多教师策略的普及，最佳教师模型往往并非最前沿模型，开源模型因更易修改反而成为更有效的蒸馏来源。

## 正文

Where I currently stand on how distillation is used and what performance uplift it uses.

For context， Anthropic HAS said that DeepSeek and others are using their models in an RL shaped data pipeline， but that does not mean it has substantial effect. These were very small numbers of samples， and likely to initialize an internal model or do a small training experiment.

The place where distillation is used as a key step is in SFT and/or midtraining （for seeding reasoning behaviors， thinking of sft and midtraining as separate is not helpful）. This is why the Chinese models say they're claude -- when training with next-token prediction， the models learn "features" that are clusters of tokens， which will regularly appear in model outputs.

This SFT data is generated by skirting the intended behavior of the API and getting reasoning tokens out. Without getting the reasoning tokens directly from the model， the data would be very hard to train on （and likely not include the "I'm claude" behaviors）.

SFT datasets have been O（1 million prompts） with high quality completions. Today， getting the prompts is often the hardest part -- especially prompts with carefully designed environments for proportional RL later， but this is mostly about SFT.

Getting to the end of it. The part where this SFT distillation is likely very useful is when a model moves into a new domain， a great example being the CripPT physics benchmark， where the US labs are far ahead. The idea would be to get some prompts， and get completions from a frontier model that you can use for an initial SFT.

With an initial SFT set， there is A TON of work to still do to get a model in the performance ballpark of GLM 5.2 and Kimi K3. The SFT stage is the start of a long， strenuous process for building a post-training recipe. It involves generating more SFT data and filtering it， something like rejection sampling， and polishing the behavior with extensive RL.

For models like Kimi K3 and GLM-5.2， the timeline is such that Fable 5 likely had no impact as a distillation teacher. That could help future models， though.

And as post-training becomes more dependent on techniques like multi-teacher on policy distillation， rather than just RL， there is even more complexity in how SFT distillation helps the final model. There are a lot of steps， and RL has been meaningfully scaled up in it's proportion of the final performance.

The real kicker for this is that in the above SFT stages， the best teacher models aren't normally the easies to integrate into the post-training recipe！ The leading fully open， reasoning SFT works （open thoughts and olmo） have had a very hard time updating their recipes to use the strongest models as teachers. Often it is a smaller， surprising model which is the best teacher. This means that it may not even be the cutting edge models that are enabling distillation！ Messy.

All together， the evidence in post-training is that distillation is becoming less impactful. Previously， before scaling RL， distillation was more impactful because the relative amount of performance gained from SFT on top of the base model was far higher.

I expect this trend to competitive， with how strong the best open weight models are -- and the teacher ambiguity clamps down on arguments that open-weight models are just "distillation washing" by removing the need to use Claude/GPT APIs. It could be that open models are just genuinely easier to distill from， by being easier to modify and tinker with.

### 引用推文

> Nathan Lambert：Yo @benthompson I'm sorry but the Chinese labs aren't using Fable / the strongest models as teachers during RL, that's not how distillation works. It wouldn't g...