New research from NVIDIA.
Does AdamW have a scale ceiling？
This work claims yes， and shows where it sits. At batch sizes up to 100M tokens for next-token prediction， SOAP and Muon maintain training stability and quality while AdamW degrades.
Higher-order optimizers have promised faster convergence for a while. The standing objection has been computational cost and numerical stability at scale.
The team identifies instabilities in SOAP at large batch sizes and eliminates the loss spikes with per-step QR orthogonalization and improved preconditioning strategies. They also measure Muon's orthogonalization quality empirically rather than assuming it.
On multi-billion-parameter models trained over trillions of tokens， both optimizers consistently beat AdamW. A layer-wise distributed optimizer compatible with Megatron-LM balances memory and hides communication without approximating the optimizer math， so the convergence benefit survives the systems layer.
Paper： https://arxiv.org/abs/2607.20548
Learn to build effective AI agents in our academy： https://academy.dair.ai/