# LLaDA MoE v2：扩散语言模型的专家混合扩展规律

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-08-04 08:00
- AIHOT 分数：46
- AIHOT 链接：https://aihot.virxact.com/items/cmsfqfs4a0a36rochayoc7s1q
- 原文链接：https://arxiv.org/abs/2608.03457

## AI 摘要

研究系统刻画了 MoE 扩散语言模型的优化超参数、计算分配与架构扩展规律，发现其与自回归模型存在定量差异。基于这些发现训练的 LLaDA MoE v2（30B-A3B）在 23.5T tokens 上从零预训练，用约 Qwen3 65% 的预训练 token 即在多项基准上接近 Qwen3，经监督微调后在八项推理与编程基准中的七项超越 SDAR Chat。

## 正文

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.