iLLaDA:改进的大型语言扩散模型
阅读原文· arxiv.orgiLLaDA是一个8B参数的掩码扩散语言模型,采用完全双向注意力机制训练,预训练数据量达12T tokens,并在25B tokens的指令数据集上微调12个epoch。模型引入变长生成与置信度评分方法以提升效率和多选评测效果。相比LLaDA,iLLaDA-Base在BBH和ARC-Challenge上分别提升21.6和14.9个点,iLLaDA-Instruct在MATH和HumanEval上分别提升14.5和16.5个点。尽管是非自回归训练,iLLaDA仍在多个基准上与Qwen2.5 7B保持竞争力。模型权重和代码已开源。
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.