# ESPO： 早停近端策略优化

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-28 08:00
- AIHOT 分数：54
- AIHOT 链接：https://aihot.virxact.com/items/cmpw15rry03fqslukyaxs4gc8
- 原文链接：https://arxiv.org/abs/2605.29860

## AI 摘要

ESPO（Early-Stopping Proximal Policy Optimization）是一种强化学习算法，它能在生成过程中实时检测失败轨迹并提前终止，以节省计算资源。该方法通过计算代理后悔值，当平滑后的累积后悔值显著超过预期时便停止生成。被截断的轨迹被视为吸收失败状态。在针对DeepSeek-R1-Distill-Qwen-7B模型的数学推理训练中，ESPO在AIME 2024、AMC 2023和MATH-500等基准上的表现均优于PPO，并累计节省了超过20%的回滚token。

## 正文

When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure noise. We propose ESPO (Early-Stopping Proximal Policy Optimization), which detects trajectory failure on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the smoothed cumulative regret significantly exceeds its estimated values. Truncated trajectories are treated as absorbing failure states with a terminal reward, concentrating negative temporal-difference (TD) errors near the detected failure step without any additional reward model or human annotation. On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME~2024 (46.28% vs. 45.25%), AMC~2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%), while saving more than 20% rollout tokens cumulatively.