Saining Xie@sainingxie · XAI 摘要
对于所有任务中稳定训练的三个最重要超参数是 lr、bs 和 beta2。随着时间推移，我们已经建立了关于如何调整它们的良好直觉，但这篇文章分析性地、令人信服地阐述了这一切。
The three biggest hps for stable training in everything are lr， bs， and beta2. We've built up good intuitions on how to tune them over time， but this lays it all out analytically and convincingly.
this is definitely my new handbook for training big models on small gpus.
🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretr...