# Nathan Lambert 课程第十讲：RL 正则化与 KL 惩罚

- 来源：Nathan Lambert (@natolambert)
- 发布时间：2026-07-29 03:34
- AIHOT 分数：36
- AIHOT 链接：https://aihot.virxact.com/items/cms52t5yt00rurorrohumzqbp
- 原文链接：https://x.com/natolambert/status/2082188013162144066

## AI 摘要

Nathan Lambert 发布其课程第十讲，主题为强化学习中的正则化。他讨论了 KL 惩罚在 RL 中不断演变的作用，并介绍了一系列优秀论文，解释为何 RL 比 SFT 能帮助模型更好地泛化，且有理论支持。课程还涉及控制奖励模型过度优化等季节性 ML 问题。

## 正文

Lecture 10 of my course！ Nominally on regularization in RL， so I discuss the evolving role of the KL penalty in RL， but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it.

When going through these， it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents.

00：00 Intro & the role of regularization
02：50 The KL penalty in RL
10：53 RL as a reverse KL loss
21：09 Why RL generalizes better than SFT
25：15 Other regularization tools

Just a few videos left as I get to the end of the course. Thanks all， and keep sending questions. Spread the word if you have a second.