# dMoE：面向扩散大语言模型的块级专家混合架构

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-29 08:00
- AIHOT 分数：63
- AIHOT 链接：https://aihot.virxact.com/items/cmpunpmki05jpslagais1162x
- 原文链接：https://arxiv.org/abs/2605.30876

## AI 摘要

扩散大语言模型（dLLMs）与混合专家（MoE）架构结合时，存在块并行解码与token级专家选择之间的不匹配，导致推理时激活专家数过多、内存占用高。为此，我们提出dMoE框架，它将每个块内的token级专家分布聚合为统一的块级专家分布，以更协调的方式指导专家路由。实验表明，dMoE将唯一激活专家数平均从69.5降至14.6，保留了原模型99.11%的性能，同时减少76.64%-79.84%的内存使用，并实现1.14至1.66倍的端到端延迟加速。代码已开源：https://github.com/fscdc/dMoE

## 正文

Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide expert routing in a more coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% of the original performance. Meanwhile, it reduces memory usage by 76.64% to 79.84% and achieves 1.14times to 1.66times end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE