SMELT:算力匹配下 MoE 循环 Transformer 的缩放律研究

HuggingFace Daily Papers(社区热门论文)·2026-09-01 08:00·2天前
AI 导读

论文提出 SMELT(Sparse MoE Transformer,中段层循环两次)配方,在逐 token FLOPs、非嵌入参数量和 KV cache 三项预算均匹配的条件下研究循环 Transformer。

HuggingFace Daily Papers(社区热门论文)
44AI 编辑部评分,满分 100

SMELT:算力匹配下 MoE 循环 Transformer 的缩放律研究

2026-09-01 08:00· 2天前
AI 导读

论文提出 SMELT(Sparse MoE Transformer,中段层循环两次)配方,在逐 token FLOPs、非嵌入参数量和 KV cache 三项预算均匹配的条件下研究循环 Transformer。

Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org