# MobileMoE：扩展设备端混合专家系统

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-26 08:00
- AIHOT 分数：67
- AIHOT 链接：https://aihot.virxact.com/items/cmpnk1omt0yazsl010oh63wqw
- 原文链接：https://arxiv.org/abs/2605.27358

## AI 摘要

MobileMoE是一系列面向设备端部署的大语言模型，采用混合专家架构，其活动参数规模为0.3-0.9B，总参数量为1.3-5.3B。该研究在移动设备内存与算力约束下，确定了“中等稀疏度结合细粒度共享专家”的最优架构设计。通过涵盖预训练、中期训练、指令微调与量化感知训练的四阶段流程，MobileMoE在14个基准测试中，以2-4倍更少的推理FLOPs达到或超越了领先的设备端密集模型性能，并以最多60%更少的参数量匹配或超过了先进的OLMoE-1B-7B模型。在商用智能手机上，其预填充和解码速度分别比密集基线MobileLLM-Pro快1.8-3.8倍和2.2-3.4倍。

## 正文

Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE, a family of on-device MoE language models with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot - moderate sparsity with fine-grained and shared experts - that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4times fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers 1.8-3.8times faster prefill and 2.2-3.4times faster decode than the dense baseline MobileLLM-Pro.