Abstract

Optimizer state is the largest single line item in the memory budget of mixture-of-experts (MoE) training: on a 6.78B-parameter MoE language model, AdamW keeps 50.6 GB of first and second moments to update 12.6 GB of bfloat16 weights. We study SkewAdam, an optimizer built on the observation that the three parameter populations of an MoE—the dense backbone, the experts, and the router—differ enough in size and gradient statistics that they should not receive the same state. SkewAdam keeps float32 momentum plus a factored second moment for the backbone (5% of parameters), a factored second moment alone for the experts (95%), and an exact second moment for the router (0.01%). The resulting state occupies 1.29 GB, 2.6% of AdamW’s, and peak training memory falls from 81.4 GB to 31.3 GB, within the budget of a 40 GB accelerator. In a controlled comparison from identical initializations over 82M tokens, SkewAdam reaches validation perplexity 108.4, ahead of AdamW (126.8), Muon (120.2), and Lion (393.7), and settles router load balance to within 1% of its uniform floor. The allocation is not what earns that perplexity: a tier ablation matches it with twenty times the state, and Adafactor, which shares the factored estimator but drops momentum, plateaus 40 points behind. The tiers buy memory at no cost to accuracy—the accuracy comes from keeping momentum, which a uniform optimizer shares too. Sweeping the baselines’ learning rates narrows but does not close the gap: the best tuned AdamW reaches 118.5, tuned Adafactor 139.7. Where optimizer state lives, these results suggest, matters at least as much as how much of it there is.

1 Introduction

Sparse mixture-of-experts (MoE) architectures decouple parameter count from per-token compute (Shazeer et al., 2017; Fedus et al., 2022; Jiang et al., 2024): the 6.78B-parameter model we study activates roughly 440M parameters per token. Optimizer state enjoys no such discount. AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2019) keeps two float32 moments for every parameter—8 bytes of state shadowing every 2-byte bfloat16 weight—so our model carries 50.6 GB of optimizer state on top of 12.6 GB of weights, and training peaks at 81.4 GB.¹¹1Throughout, GB denotes GiB ( bytes), matching torch.cuda.max_memory_allocated. The component that owns most of those parameters, the expert bank, is also the one whose individual parameters are touched least often.

Memory-efficient optimizers exist, but they treat the network as a homogeneous block. Lion (Chen et al., 2023) keeps one momentum buffer and updates with its sign, halving state but discarding gradient magnitude everywhere, including in the router, where relative magnitudes carry the load-balancing signal. Muon (Jordan et al., 2024) also keeps a single buffer and orthogonalizes each update by Newton–Schulz iteration, a structural prior designed for dense hidden layers. Adafactor (Shazeer and Stern, 2018) factors the second moment into row and column statistics, which suits a 16.8M-parameter expert matrix well but applies the same recipe to the 0.5M-parameter router that decides where every token goes. None of these methods asks whether different parts of an MoE deserve different state.

SkewAdam does. It allocates optimizer state by tier (Figure 1). The dense backbone—embeddings, attention, the dense feed-forward block—holds 5% of parameters and sees every token, so float32 momentum is cheap there and earns its keep; it also gets a factored second moment. The experts hold 95% of parameters, but under top-2 routing over 128 experts each one processes on average of the tokens; they keep only a factored second moment, which removes the single largest state cost in the model. The router keeps an exact, unfactored second moment: per-logit adaptivity costs 2 MB and steers all the traffic. The name is the design—the state budget is skewed toward where it pays.

We make three contributions. First, we describe the tiered allocation policy and its closed-form memory model: 1.29 GB of optimizer state on a 6.78B-parameter MoE, with peak training memory of 31.3 GB. Second, we run a controlled single-GPU comparison against AdamW, Lion, and Muon—identical initialization, identical data order, and an identical bfloat16 weight-update path with dithered stochastic rounding for all four—in which SkewAdam attains the best validation perplexity (108.4) at a throughput within 1.5% of the fastest baseline; same-protocol follow-ups on two further GPUs add Adafactor and a GaLore-style baseline, ablate the policy’s tiers, and sweep the baselines’ learning rates: the policy matches full-momentum perplexity at a twentieth of the state, and its lead over the baselines survives their tuning. Third, we analyze routing stability by measuring the load-balancing loss against its analytic floor, finding that Lion’s magnitude-blind updates cost perplexity rather than balance, and that Muon drifts away from balance late in training.

媒体内容 · 前往原文查看

2 Related work

Memory-efficient optimizers.

Adafactor (Shazeer and Stern, 2018) is the closest relative of this work. The factored estimator inside SkewAdam is Adafactor’s nonnegative rank-one estimator written with row and column means instead of sums, and the update-RMS clipping is Adafactor’s as well; we claim no novelty for either. What we take up is the allocation question that uniform recipes leave open: Adafactor either drops momentum everywhere or keeps it everywhere, and factors every matrix regardless of its role. SM3 (Anil et al., 2019) and 8-bit optimizers (Dettmers et al., 2022) compress state by other means, and ZeRO (Rajbhandari et al., 2020) shards it across devices rather than shrinking it; all are compatible with, and orthogonal to, a tiered policy. GaLore (Zhao et al., 2024) projects gradients into a low-rank subspace before applying Adam, again uniformly across matrices.

Sign- and geometry-based updates.

Lion (Chen et al., 2023) reduces state to a single buffer by updating with the sign of an interpolated momentum. Muon (Jordan et al., 2024) replaces adaptive scaling with Newton–Schulz orthogonalization of the momentum and has been scaled to large dense and MoE models (Liu et al., 2025); its authors route embeddings and other non-matrix parameters to an Adam-style rule, which our implementation follows. Both methods are uniform over hidden matrices, expert or not.

MoE training.

Sparsely gated MoE layers were introduced by Shazeer et al. (2017) and scaled by GShard (Lepikhin et al., 2021) and Switch Transformers (Fedus et al., 2022), whose load-balancing auxiliary loss we use. The router z-loss follows ST-MoE (Zoph et al., 2022), which documents how fragile router training can be; Mixtral (Jiang et al., 2024) and DeepSeekMoE (Dai et al., 2024) are recent open systems. This literature concentrates on architecture and losses; optimizer state allocation for MoE has received little direct attention.

Low-precision training.

Mixed-precision practice keeps float32 master weights (Micikevicius et al., 2018); pure-bfloat16 training is attractive for memory but loses small updates to rounding (Kalamkar et al., 2019). Stochastic rounding repairs much of the damage (Gupta et al., 2015; Zamirai et al., 2021). We train with bfloat16 master weights and a dithered approximation to stochastic rounding (Section 3), applied identically under every optimizer we compare.

3 SkewAdam: tiered state allocation

Factored second moments.

For a weight matrix with gradient , SkewAdam maintains exponential moving averages of the row-wise (, over columns) and column-wise (, over rows) mean squared gradient,

(1)

and reconstructs the second-moment matrix as the rank-one estimate , where is the mean of . This is exact whenever the true second-moment matrix has rank one and coincides with Adafactor’s estimator (Shazeer and Stern, 2018). Storage falls from to floats per matrix; for a expert matrix, from 64 MB to 32 KB.²²2Stacked expert tensors of shape factor along the last two axes; in our model the experts are separate matrices, so the two-dimensional path is the one exercised.

The tiers.

The policy assigns state by parameter role (Figure 1, Algorithm 1). Backbone tensors (embeddings, attention projections, the dense feed-forward block, norms) carry float32 momentum and a factored second moment. Their gradients are dense—every token contributes every step—so momentum smooths a signal that is actually there, and at 5% of parameters the buffer costs 1.27 GB. Expert tensors carry only the factored second moment. They are 95% of parameters, each expert sees roughly of tokens under top-2-of-128 routing, and a momentum buffer here would cost 24 GB to smooth gradients that arrive sparsely and with high variance; Adafactor’s results suggest momentum can be dropped without losing adaptivity (Shazeer and Stern, 2018). Router weights keep a full, unfactored second moment and no weight decay. Factoring a gate would pool scale statistics across the very logits whose relative magnitudes determine load balance; exactness here costs 2 MB. The policy is a judgment about where each ingredient of Adam earns its memory, not a theorem; Section 5 tests it.

Update clipping and low-precision updates.

The preconditioned update is clipped to unit root-mean-square, , Adafactor’s update clipping with threshold . Master weights are bfloat16. Each step is computed in float32 and written back through a dithered rounding: uniform noise of one-ULP width, with , is added before the cast, approximating unbiased stochastic rounding (Gupta et al., 2015; Zamirai et al., 2021). Every optimizer in our comparison uses this same write-back path, so none is privileged by precision handling.³³3A side effect we accept deliberately: with and , the decoupled weight-decay step changes weights by relative—more than two orders of magnitude below the bfloat16 ULP of —so weight decay rounds to a no-op in all runs. The comparison is fair but effectively unregularized; see Section 6.

Memory model.

Let , , be the tier sizes. SkewAdam’s state is

(2)

since the factored vectors are negligible. With M, M, M this gives 1.29 GB (a component-level account is in Appendix B), against  GB for AdamW. The saving is structural: it does not depend on quantization and would compound with it.⁴⁴4Quantized state also carries a scale ceiling that factoring avoids. Current 8-bit optimizer kernels terminate the process—a hard exit from C++, not an exception—once a single parameter tensor reaches elements, a size that fused expert weights reach in larger MoE models (bitsandbytes issue 1785). We measured the boundary on an A100: the 8-bit step succeeds at elements and kills the process at , while factored state, built from native PyTorch ops with 64-bit indexing, crosses it cleanly. Scripts and logs are in the repository’s experiments/ directory.

媒体内容 · 前往原文查看

1: in float32

2:if or is a vector then

3:  ;   full second moment

4:else

5:  update by Eq. (1);   factored, with -clamps

6:end if

7:if then

8:  ;   fp32 momentum

9:else

10:   no momentum buffer

11:end if

12: update clipping

13:,   dithered rounding

4 Experimental setup

Model.

We train a decoder-only transformer with two blocks: the first uses a dense SwiGLU feed-forward layer (Shazeer, 2020), the second an MoE layer with 128 SwiGLU experts of hidden width 4096 and top-2 routing. Width is 4096 throughout, with grouped-query attention (32 query heads, 8 KV heads) (Ainslie et al., 2023), learned positions, a GPT-2 BPE vocabulary padded to 50,304 (Radford et al., 2019), and tied embeddings; 6,784M parameters in total, about 440M active per token. Two blocks is shallow by intent as well as by budget: it concentrates 95% of parameters in a single expert bank, the population whose optimizer state we want to stress, and it lets a 6.78B-parameter, four-optimizer comparison run on one GPU. The router operates in float32 with input noise at training time and a zero-initialized gate; we use the Switch-style balancing loss with (Fedus et al., 2022) and a z-loss of (Zoph et al., 2022). LayerNorms are kept in float32.

Data and protocol.

We stream OpenWebText (Gokaslan and Cohen, 2019), hashing each document into a 95/5 train/validation split so the two sides share no documents. Each run takes 10,000 steps at batch size 64 128 tokens (81.9M tokens, single epoch; no batch repeats). Validation uses the same 64 held-out batches (0.5M tokens) for every optimizer. All four optimizers start from one shared initialization and consume identical batches in identical order, under bfloat16 autocast with per-block activation checkpointing (Chen et al., 2016) and gradient clipping at 1.0. Learning rates follow a cosine schedule with 3% warmup: for AdamW and SkewAdam, for Lion (the – reduction its authors recommend), and for Muon’s matrices with an internal Adam at for embeddings, router, and vector parameters. Runs execute on a single NVIDIA H200; its 141 GB simply lets us include the AdamW baseline that a 40 GB device could not hold. Peak memory is read from CUDA’s peak-allocation counter. Full hyperparameters are in Appendix A.

5 Results

媒体内容 · 前往原文查看

Optimizer	State (GB)	Peak mem. (GB)	Tokens/s	Val. PPL	Balance loss
SkewAdam	1.29	31.3	5,000	108.4	0.0505
AdamW	50.55	81.4	4,692	126.8	0.0502
Muon	25.27†	57.6	3,409	120.2	0.0608
Lion	25.27	56.6	5,075	393.7	0.0537
†One float32 buffer per parameter; the Adam-style state Muon keeps for embeddings, router, and
†vector parameters adds roughly 0.8 GB.

Memory and throughput.

Table 1 and Figure 2b give the budget. AdamW peaks at 81.4 GB, of which 50.6 GB is optimizer state; the rest is bfloat16 weights (12.6 GB), gradients (12.6 GB), and activations. Lion and Muon halve the state and still peak near 57 GB, above the 40 GB class of accelerators. SkewAdam’s 1.29 GB of state brings the peak to 31.3 GB, with headroom to spare on a 40 GB device. Throughput tracks state traffic: SkewAdam sustains 5,000 tokens/s, 6.6% above AdamW, and 1.5% below Lion, whose single buffer is the cheapest to maintain. Muon pays 32% relative to SkewAdam for running Newton–Schulz iterations over 6.4B expert parameters every step.

Convergence.

AdamW and Muon converge faster for the first 3,000 steps—at step 1,000 SkewAdam trails AdamW by 114 points of perplexity—but SkewAdam passes both by step 4,000 and ends at 108.4 against 120.2 (Muon) and 126.8 (AdamW); per-step values are tabulated in Appendix C. Lion ends at 393.7, more than three times SkewAdam, having peaked at 381.1 at step 9,000 and worsened thereafter. Lion does not recover. We read the Lion result as consistent with the magnitude-blindness account—a fixed-magnitude step treats a heavily routed expert and a rarely routed one identically—while noting it reflects one learning rate and one seed. That SkewAdam ends below AdamW is the more surprising outcome, and we offer an interpretation rather than a claim: with 128 tokens reaching each expert per step, Adam’s per-coordinate second moments are estimated from little data, while the factored estimator pools statistics over 4096 coordinates per row and column; the pooled preconditioner, plus RMS clipping of occasional large updates, may simply be better conditioned at this routing sparsity.

媒体内容 · 前往原文查看

Optimizer	State (GB)	Peak mem. (GB)	Tokens/s	Val. PPL	Balance loss
SkewAdam	1.29	31.3	3,778	109.0	0.0505
Adafactor	0.01	29.6	4,032	149.5	0.0502
GaLore-style (rank 128)	—	31.7	4,209	1,839.9	0.0510

Adafactor and GaLore.

The comparison this method most owes its readers is against Adafactor, which supplies the factored estimator SkewAdam builds on. We ran it when compute later allowed: same code, data, seed, and shared initialization, on an H100 MIG slice (Table 2). SkewAdam, re-run in the same batch as the anchor, lands at 109.0 against its 108.4 on the H200—0.5% apart, so the protocol transfers across hardware. Uniform Adafactor carries even less state than SkewAdam (12 MB; no momentum anywhere) but plateaus at 149.5, forty perplexity points behind, its final thousand steps improving by less than 0.1. Both optimizers share the rank-one second-moment estimator and update clipping; the difference is that SkewAdam keeps momentum while Adafactor drops it entirely, alongside Adafactor’s annealed decay.⁵⁵5HuggingFace Adafactor anneals its second-moment decay as rather than holding . The tier ablation below (Table 3) locates this gap in the momentum and its decay schedule, not the allocation: uniform allocation with momentum reaches the same perplexity as the tiered policy. The rank-128 GaLore-style baseline in our trainer fails outright at these settings (1,839.9), balancing router load, like Lion, while never learning the language model. We read this as a caution about projecting sparse expert gradients onto low-rank subspaces, not as a verdict on GaLore, whose reference implementation and tuning differ from ours.

Which tier does the work?

Table 3 toggles the tiers one at a time, on the MI300X under the same protocol. Every variant reaches the same validation perplexity (108.2–108.9, within single-seed noise) and the same load balance (); what moves twentyfold is optimizer state. Restoring momentum to the experts costs 24 GB and moves perplexity by 0.2—expert momentum is dead weight, which is exactly what the policy discards. Factoring the router changes neither perplexity nor balance, so the exact router (2 MB) is a harmless but not load-bearing choice. Uniform allocation, with momentum and factoring everywhere, matches the tiered policy at twenty times the state. The policy’s contribution is therefore memory: it recovers full-momentum perplexity from momentum on the dense backbone alone, where it costs 1.27 GB rather than 25. SkewAdam itself reaches 108.9 here, matching its 108.4 (H200) and 109.0 (H100) numbers—a third platform, and a second vendor, within noise.

媒体内容 · 前往原文查看

Variant	State (GB)	Peak mem. (GB)	Val. PPL	Balance loss
SkewAdam (full policy)	1.29	31.4	108.9	0.0505
+ momentum on experts	25.29	55.4	108.7	0.0506
factored router	1.28	31.4	108.2	0.0505
uniform (momentum + factored)	25.29	55.4	108.3	0.0503

Tuning the baselines.

The one quality claim left standing after the ablation—SkewAdam ahead of AdamW—rested on a single untuned learning rate, so we swept both strong baselines while leaving SkewAdam at its default (Table 4). Tuning is worth real perplexity to them: AdamW improves from 126.8 at to at (three seeds), Adafactor from 149.5 to 139.7 at the same rate (two seeds, 0.005 apart). Adafactor’s minimum is bracketed on both sides— undertrains to 257.5 and every higher rate is worse—while AdamW’s is bracketed only from above, though Adafactor’s collapse at suggests this step budget undertrains at lower rates generally. Neither tuned baseline reaches SkewAdam: untuned at , it leads the best AdamW by ten perplexity points, twenty times the seed-level standard deviation, and the best Adafactor by thirty-one. Tuning narrows the headline gaps; it does not close them. The persistent AdamW–Adafactor separation (118.5 vs 139.7, both tuned) is the momentum story of Table 3 again, now visible between independent optimizers.

媒体内容 · 前往原文查看

Optimizer	LRs swept	Best LR	Best Val. PPL
AdamW	, ,		(3 seeds)
Adafactor	, , , ,		139.7 (2 seeds)
SkewAdam (untuned)	—		108.4–109.0 (3 GPUs)

Routing stability.

The balancing loss equals exactly when both the router probabilities and the realized token assignment are uniform, which makes deviation from a calibrated imbalance measure (Figure 3). SkewAdam and AdamW stay within 1% of the floor from step 4,000 onward (final values 0.0505 and 0.0502). Lion is stable but elevated 7–9% above the floor throughout—it balances load while failing to learn the language model. Muon improves steadily to 1–3% above floor and then, in the last thousand steps, jumps to 0.0608, 22% above. Whether this is the onset of an instability or a transient cannot be settled from one run, but no other optimizer moves by that much at any point in training, and the jump coincides with Muon’s only flat segment in validation perplexity.

Zero-shot evaluation.

For completeness we score the final checkpoints with the LM Evaluation Harness (Gao et al., 2023) on PIQA, WinoGrande, HellaSwag, and ARC-Challenge (Bisk et al., 2020; Sakaguchi et al., 2020; Zellers et al., 2019; Clark et al., 2018). After 82M tokens—three to four orders of magnitude below modern budgets—all four models sit near chance on three of the four tasks and a few points above chance on PIQA (53.5–55.9%), with no separation between optimizers beyond one to two standard errors (full table in Appendix D). At this token budget the downstream numbers neither support nor undermine any optimizer; we report them so that the perplexity gains are not mistaken for more than they are.

6 Limitations

The model is two blocks deep. That choice concentrates 95% of parameters in one expert bank, which is the stress test we wanted, but it leaves untested how tiered allocation behaves when routing decisions compose across many MoE layers. Most numbers are one run per configuration. The sweeps of Table 4 addressed this where it mattered most—both strong baselines are tuned over bracketing grids with repeated seeds, and SkewAdam’s own number is replicated on three GPUs—but SkewAdam was not itself tuned, AdamW was not probed below , and Lion and Muon keep single untuned rates. The 82M-token horizon and 128-token contexts are small, and weight decay was inert in all runs (Section 3); anyone scaling this recipe to production horizons must reintroduce weight decay, fusing the decay term into the float32 update ahead of the stochastically rounded cast so that it survives the bfloat16 ULP, a configuration we have not yet validated at length. The Adafactor and GaLore runs of Table 2 were added when compute later became available; they share everything with the main protocol except the GPU; where configurations repeat across machines they agree closely (SkewAdam within 0.6 perplexity over three GPUs, AdamW at within 0.5 over two). The GaLore number in particular reflects a single untuned configuration of our own implementation and should not be read as more than that. The tier ablation (Table 3) likewise rests on one seed per variant, and its perplexity spread (0.6 across four variants) is within run-to-run noise; we read it as evidence of memory-at-parity, not of any perplexity ordering among the variants. It leaves open whether the tiers separate at all under a multi-seed protocol, or at a longer horizon where expert momentum might begin to earn its 24 GB. Finally, the downstream evaluations are near chance and should be read as a completeness check, not evidence of capability. We plan to validate the approach at larger scales as compute becomes available.

7 Conclusion

A mixture-of-experts model is not a homogeneous bag of parameters, and its optimizer need not pretend otherwise. Spending Adam’s full state only on the dense backbone, factored variance on the expert bank, and an exact second moment on the router cuts optimizer state by 97.4% and peak training memory by 61% on a 6.78B-parameter MoE. A tier ablation shows this costs nothing: the policy reaches the same perplexity and routing balance as a uniform optimizer carrying twenty times the state. The contribution is memory, not a better optimizer—Adam-family quality at 2.6% of Adam’s state, a comparison that survives sweeping the baselines’ learning rates. The broader suggestion is a design principle rather than a single optimizer: decide where optimizer state lives tier by tier, with the gradient statistics of each tier in view.

Acknowledgments

This work was self-funded and run on rented GPU time; the compute budget, not the experimental design, set the scale of the study. The author welcomes collaboration or compute support to extend the comparison to deeper models, longer horizons, and multi-seed replication.

References

• J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai (2023) GQA: training generalized multi-query transformer models from multi-head checkpoints. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §4.
• R. Anil, V. Gupta, T. Koren, and Y. Singer (2019) Memory efficient adaptive optimization. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.
• Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi (2020) PIQA: reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, Cited by: §5.
• T. Chen, B. Xu, C. Zhang, and C. Guestrin (2016) Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Cited by: §4.
• X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C. Hsieh, Y. Lu, and Q. V. Le (2023) Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.
• P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §5.
• D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066. Cited by: §2.
• T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer (2022) 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations (ICLR), Cited by: §2.
• W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), pp. 1–39. Cited by: §1, §2, §4.
• L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, et al. (2023) A framework for few-shot language model evaluation. Note: https://github.com/EleutherAI/lm-evaluation-harness Cited by: §5.
• A. Gokaslan and V. Cohen (2019) OpenWebText corpus. Note: http://Skylion007.github.io/OpenWebTextCorpus Cited by: §4.
• S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan (2015) Deep learning with limited numerical precision. In International Conference on Machine Learning (ICML), pp. 1737–1746. Cited by: §2, §3.
• A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, et al. (2024) Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §1, §2.
• K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein (2024) Muon: an optimizer for hidden layers in neural networks. Note: https://kellerjordan.github.io/posts/muon/ Cited by: §1, §2.
• D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, et al. (2019) A study of BFLOAT16 for deep learning training. arXiv preprint arXiv:1905.12322. Cited by: §2.
• D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: §1.
• D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen (2021) GShard: scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), Cited by: §2.
• J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, et al. (2025) Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982. Cited by: §2.
• I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: §1.
• P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu (2018) Mixed precision training. In International Conference on Learning Representations (ICLR), Cited by: §2.
• A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. OpenAI Technical Report. Cited by: §4.
• S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He (2020) ZeRO: memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), Cited by: §2.
• K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi (2020) WinoGrande: an adversarial winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence, Cited by: §5.
• N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.
• N. Shazeer and M. Stern (2018) Adafactor: adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning (ICML), pp. 4596–4604. Cited by: §1, §2, §3, §3.
• N. Shazeer (2020) GLU variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §4.
• P. Zamirai, J. Zhang, C. R. Aberger, and C. De Sa (2021) Revisiting BFloat16 training. arXiv preprint arXiv:2010.06192. Cited by: §2, §3.
• R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi (2019) HellaSwag: can a machine really finish your sentence?. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §5.
• J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian (2024) GaLore: memory-efficient LLM training by gradient low-rank projection. In International Conference on Machine Learning (ICML), Cited by: §2.
• B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus (2022) ST-MoE: designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906. Cited by: §2, §4.

Appendix A Hyperparameters

媒体内容 · 前往原文查看

Model		Training
Width	4096	Steps	10,000
Blocks	2 (dense FFN, MoE)	Tokens	81.9M (single epoch)
Attention heads / KV heads	32 / 8	Batch	64 seq 128 tokens
Dense FFN hidden (SwiGLU)	4096	Microbatch	8 (8 accumulation steps)
Experts / hidden / top-	128 / 4096 / 2	Gradient clip	1.0
Router noise (train)		Schedule	cosine, 3% warmup
Balance coef. / z-loss	0.05 /	Seed	42
Vocabulary (GPT-2 BPE, padded)	50,304	Precision	bf16 master weights
Positions (learned, max)	256		fp32 norms and router
Dropout	0	Checkpointing	per block
Optimizers
AdamW	, ,
SkewAdam	, ,
Lion	,
Muon	, momentum , 3 NS steps; internal Adam
Adafactor	HF impl.; , no momentum, decay
GaLore-style	rank 128, , ; full Adam off-matrix
Weight decay	0.05 (0 on router); inert under bf16 rounding, see Section 3

Appendix B Memory accounting

媒体内容 · 前往原文查看

Component	Parameters	State kept	Size
Backbone momentum (fp32)	341.4M		1,302.2 MB
Backbone factored 2nd moment	—	vectors	0.5 MB
Backbone vector params (norms)	0.04M	full	0.2 MB
Expert factored 2nd moments	6,442.5M	per matrix	12.0 MB
Router full 2nd moment	0.52M		2.0 MB
Total	6,784.3M		1.29 GB
AdamW on the same model	6,784.3M		50.55 GB

Parameter counts by tier: token embedding M (tied with the output head), positions M, attention M, dense FFN M, norms M, for a backbone of M; experts M; router M. Total M, matching the trainer’s logged count.

Appendix C Convergence detail

Table 7 tabulates validation perplexity at every evaluation step; Figure 4 shows the training loss traces and final sustained throughput. Muon leads for the first three thousand steps, SkewAdam from step four thousand on.

媒体内容 · 前往原文查看

Step ()	1	2	3	4	5	6	7	8	9	10
SkewAdam	478.4	287.6	210.4	172.8	153.2	134.9	120.9	113.7	108.9	108.4
AdamW	364.1	268.9	208.4	181.3	167.4	150.8	138.8	132.3	127.5	126.8
Muon	316.9	234.8	198.5	178.6	162.0	146.4	132.9	125.5	120.9	120.2
Lion	812.0	747.8	671.2	552.7	507.9	466.9	421.7	397.4	381.1	393.7

媒体内容 · 前往原文查看

Step ()	1	2	3	4	5	6	7	8	9	10
SkewAdam	477.3	293.8	211.9	174.0	155.1	135.7	121.6	114.4	109.6	109.0
Adafactor	403.2	287.0	242.7	207.3	187.1	169.2	157.8	152.4	149.6	149.5
GaLore	2703.1	2256.8	2070.4	1965.6	1901.7	1855.1	1829.0	1821.7	1826.0	1839.9

Appendix D Zero-shot evaluation detail

Table 9 expands the summary in Section 5. No pairwise difference between optimizers exceeds two standard errors on any task.

媒体内容 · 前往原文查看

Optimizer	PIQA	WinoGrande	HellaSwag	ARC-Challenge
SkewAdam	54.7 1.2	49.3 1.4	25.4 0.4	21.6 1.2
AdamW	54.6 1.2	49.4 1.4	25.3 0.4	22.9 1.2
Muon	55.9 1.2	50.4 1.4	25.4 0.4	22.0 1.2
Lion	53.5 1.2	51.1 1.4	25.3 0.4	22.2 1.2

Appendix E Reproducibility

The main comparison ran on a single NVIDIA H200 (141 GB) using the released single-file trainer and evaluator. Code, per-step training logs, and figures are available at https://github.com/nuemaan/skewadam. The environment is installed with

pip install torch numpy transformers bitsandbytes datasets lm_eval

and the four reported runs come from one invocation that trains all four optimizers in sequence:

CUDA_VISIBLE_DEVICES=0 python train.py \
    --optimizers "skewadam,adam,lion,muon"
python evaluate.py runs/best_skewadam.pt   # likewise for the others
python plot_metrics.py

The trainer builds one initialization and one cached batch sequence, reseeds before each optimizer, and reuses both for all four, which is what licenses the matched-conditions claim of Section 4.

The follow-up of Table 2 ran on a different machine (NVIDIA H100 NVL, 47 GB MIG slice) under the same protocol:

python train.py --optimizers "adafactor,skewadam,galore" \
    --dataset-name Skylion007/openwebtext

The dataset flag points at the canonical parquet mirror of the same corpus (newer releases of the datasets library no longer accept the legacy openwebtext id); the document-hash split is unchanged, so the validation set is identical. Metrics and the training log for this run are kept under runs/h100/. The tier ablation and the learning-rate sweeps ran on an MI300X via experiments/tier-ablation/run_ablation.py and experiments/lr-sweep/run_sweep.py / run_fix.py; per-run metrics are under runs/amd-ablation/ and runs/lr-sweep/. The trainer writes per-step metrics, analytic state sizes, and peak memory to JSON files under runs/; every number in this paper is read from those files or from the evaluation JSONs. The data split is deterministic (an MD5 hash of each document’s first kilobyte selects train or validation), so the validation set is identical across runs and machines.