Nathan Lambert@natolambert

2026-07-29 03:34· 32分钟前

AI 摘要

Nathan Lambert 发布其课程第十讲，主题为强化学习中的正则化。他讨论了 KL 惩罚在 RL 中不断演变的作用，并介绍了一系列优秀论文，解释为何 RL 比 SFT 能帮助模型更好地泛化，且有理论支持。课程还涉及控制奖励模型过度优化等季节性 ML 问题。

Lecture 10 of my course！ Nominally on regularization in RL， so I discuss the evolving role of the KL penalty in RL， but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it.

When going through these， it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents.

00：00 Intro & the role of regularization 02：50 The KL penalty in RL 10：53 RL as a reverse KL loss 21：09 Why RL generalizes better than SFT 25：15 Other regularization tools

Just a few videos left as I get to the end of the course. Thanks all， and keep sending questions. Spread the word if you have a second.

教程/实践数据/训练

在 X 查看原推

Nathan Lambert@natolambert · X

36导出 Markdown

2026-07-29 03:34·32分钟前

在 X 看原推· x.com

AI 摘要

When going through these， it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents.

00：00 Intro & the role of regularization 02：50 The KL penalty in RL 10：53 RL as a reverse KL loss 21：09 Why RL generalizes better than SFT 25：15 Other regularization tools

Just a few videos left as I get to the end of the course. Thanks all， and keep sending questions. Spread the word if you have a second.