# Switch-KD：面向视觉-语言模型的视觉切换知识蒸馏

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-16 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo2o8b6903hnslba45mht5if
- 原文链接：https://arxiv.org/abs/2604.14629

## AI 摘要

研究团队提出 Switch-KD 视觉切换知识蒸馏框架，通过将学生模型的视觉输出接入教师模型的语言路径，在共享文本概率空间中实现跨模态知识迁移。该方法包含动态双向 Logits 差分损失函数，可自适应对齐关键概率区域并保持分布结构。实验表明，0.5B 参数的 TinyLLaVA 在无需修改架构的情况下，从 3B 教师模型蒸馏知识后，在 10 个多模态基准测试中平均性能提升 3.6 个百分点。

## 正文

Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.