# 一步梯度延迟并非大规模异步流水线并行LLM预训练的障碍

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-29 08:00
- AIHOT 分数：51
- AIHOT 链接：https://aihot.virxact.com/items/cmr0qq31y047hslolskrh74du
- 原文链接：https://arxiv.org/abs/2606.30634

## AI 摘要

异步流水线并行消除同步气泡，但梯度陈旧问题限制其应用。PipeDream-2BW调度保证恒定一步延迟，但业界认为该陈旧性下优化不稳定。本文通过实证分析推翻此假设，发现AdamW在一步延迟下退化，而Muon等优化器鲁棒性强。研究进一步提出优化器无关的误差反馈修正以缓解延迟影响。在10B参数模型上的实验表明，所提策略可缩小异步与同步训练的性能差距，展示大规模异步流水线并行的实用潜力。

## 正文

Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational resources. Asynchronous Pipeline Parallelism eliminates these bubbles, maximizing throughput at the cost of gradient staleness. Among asynchronous schedules, PipeDream-2BW is particularly appealing: unlike the original PipeDream schedule, it ensures a constant one-step gradient delay regardless of pipeline depth. However, its adoption remains limited due to the common belief that optimizing under staleness is fundamentally unstable. In this work, we challenge this assumption, demonstrating that degradation under one-step delay depends strongly on optimizer choice rather than being an intrinsic limitation. We provide the first comprehensive empirical analysis showing that while AdamW, the predominant optimizer at the time when PipeDream-2BW was introduced, indeed suffers from severe degradation, recent methods like Muon exhibit strong robustness under a one-step delay. We introduce an optimizer-agnostic Error Feedback-inspired correction to further mitigate delay effects. We provide supporting theoretical analysis demonstrating convergence for Muon with and without this correction. Extensive evaluation on models up to 10B parameters confirms that our strategies bridge the performance gap with synchronous training, highlighting the practical potential of asynchronous pipeline parallelism at scale.