# Mix-MoE：通过混合 MoE 改进大语言模型的多语言机器翻译

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-23 08:00
- AIHOT 分数：57
- AIHOT 链接：https://aihot.virxact.com/items/cmply5bi10jv3sl01oygn002z
- 原文链接：https://arxiv.org/abs/2605.24681

## AI 摘要

Mix-MoE 框架通过两阶段训练改进大语言模型的多语言翻译能力：先在单语语料上用混合专家进行后预训练，再在平行语料上训练。它将专家层分为语言模型专家和机器翻译专家，分别保留单语知识和学习翻译知识，并引入基于傅里叶变换特征的路由机制促进专家交互。实验表明，该框架在多语言翻译任务上显著优于现有基线，有效缓解了参数干扰问题。

## 正文

Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT), even with limited bilingual supervision. However, fine-tuning LLMs with parallel corpora presents major challenges, namely parameter interference. To address these issues, we propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT. Our framework operates in two distinct stages: (1) post-pretraining with MoE on monolingual corpora, and (2) post-pretraining with MoE on parallel corpora. Crucially, we divide the MoE layers into two specialized groups: Language Model Experts (LM Experts) and Machine Translation Experts (MT Experts). LM Experts are designed to capture and retain the monolingual knowledge learned by the pre-trained LLM. MT Experts, on the other hand, are specifically trained to acquire and store bilingual translation knowledge. Furthermore, to facilitate effective interaction between these specialized experts and leverage potential underlying structural patterns in text, we introduce a routing mechanism enhanced by Fourier Transform features derived from model representations. The experimental results demonstrate that Mix-MoE excels in multilingual MT, significantly outperforming existing baselines and showing notable progress in mitigating parameter interference.