# CEPO：基于对比证据的策略优化方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-19 08:00
- AIHOT 分数：65
- AIHOT 链接：https://aihot.virxact.com/items/cmpdf6znh03ehslk1997m4u08
- 原文链接：https://arxiv.org/abs/2605.19436

## AI 摘要

针对强化学习（RLVR）中对所有生成token给予相同奖励导致的信用分配问题，本文提出对比证据策略优化（CEPO）。该方法通过对比正确与错误答案对每个token的偏好，精准区分关键推理步骤与普通填充token，从而获得更细粒度的奖励信号。CEPO利用训练批次中已有的拒绝样本构建错误答案教师模型，不产生额外采样成本。实验表明，在2B和4B参数规模下，CEPO在多个数学推理基准上分别取得43.43%和60.56%的平均准确率，优于GRPO等基线方法。

## 正文

When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler. A natural fix is to condition the model on the correct answer as a teacher, identifying tokens it would have generated differently had it known the answer. Prior work shows this either corrupts training by leaking the answer into the gradient, or produces a weak signal that cannot distinguish decisive steps from filler, since both look equally surprising relative to the model's baseline. We propose Contrastive Evidence Policy Optimization (CEPO), which asks a sharper question at every token: not just "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?" A token satisfying both is a genuine reasoning step; one satisfying neither is filler. The wrong-answer teacher is constructed from rejected rollouts already in the training batch, incurring no additional sampling cost. We prove CEPO inherits all structural safety guarantees of the prior state of the art while strictly sharpening credit at decisive tokens, with the improvement vanishing exactly at filler positions. Empirically, CEPO achieves 43.43% and 60.56% average accuracy across five multimodal mathematical reasoning benchmarks at 2B and 4B scale, respectively, versus 41.17% and 57.43% for GRPO under identical training budgets. Distribution-matching self-distillation methods (OPSD, SDPO) fall below the untrained baseline, empirically confirming the information leakage our theory predicts. Our code is available at https://github.com/ahmedheakl/CEPO.