重新思考扩散模型Transformer中的跨层信息路由
阅读原文· arxiv.org扩散模型的主流架构Diffusion Transformers (DiTs)沿用了原始Transformer的残差连接。本文通过系统性分析发现,这种传统残差加法在模型深度和去噪时间步的联合维度上存在信息幅度膨胀、梯度衰减和冗余等问题。为此,研究者提出了即插即用的扩散自适应路由(DAR)作为替代方案,它通过可学习的机制对子层输出历史进行时间步自适应的聚合。在ImageNet 256×256实验中,DAR将SiT-XL/2的FID分数从9.67提升至7.56,并减少了达到基线收敛质量所需的训练迭代。该方法还可与REPA等兼容以加速训练,并应用于文生图模型的微调。
Diffusion Transformers (DiTs) have become a de facto backbone of modern visual generation, and nearly every major axis of their design -- tokenization, attention, conditioning, objectives, and latent autoencoders -- has been extensively revisited. The residual stream that governs how information accumulates across layers, however, has been directly inherited from the original Transformer. In this paper, we present a systematic empirical analysis of cross-layer information flow in DiTs, jointly along depth and denoising timestep, and identify three concrete symptoms of traditional residual addition, namely monotonic forward magnitude inflation, sharp backward gradient decay, and pronounced block-wise redundancy. Motivated by this diagnosis, we propose Diffusion-Adaptive Routing (DAR), a drop-in residual replacement that performs learnable, timestep-adaptive, and non-incremental aggregation over the history of sublayer outputs. Moreover, the proposed DAR is compatible with many modern Transformer enhancement methods, such as REPA. On ImageNet 256times256, DAR improves SiT-XL/2 by 2.11 FID (7.56 vs.\ 9.67) and matches the baseline's converged quality with 8.75times fewer training iterations. Stacked on top of REPA, it yields a 2times training acceleration in the early stage, suggesting cross-layer information routing as an underexplored design axis in diffusion modeling, one that operates orthogonally to existing representation-alignment objectives. Beyond pretraining, DAR can also be applied during the fine-tuning stage of large-scale T2I models and preserves high-frequency details during Distribution Matching Distillation.