# 多模态预训练的物理机制：知识流动、模态协同、早期统一与训练配方

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-08-05 08:00
- AIHOT 分数：55
- AIHOT 链接：https://aihot.virxact.com/items/cmsgxbipg02rlroxzmu8zwu7f
- 原文链接：https://arxiv.org/abs/2608.05000

## AI 摘要

一项系统研究揭示多模态预训练的内在机制：语言、视觉理解与视觉生成在跨模态知识传递中呈现不同影响模式与不对称性；数据“复杂度”决定模态协同与否，共享注意力与归一化加模态特定前馈层可促进协同。从极早期统一模态联合训练优于后期对齐或顺序训练，但延迟整合会引发“视觉惰性”现象。该研究仅用5%计算预算即可实现强生成性能，并通过训练多个13.5B MoE模型、2T tokens验证了结论。

## 正文

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.