# C2：基于二元偏好的可扩展评分标准增强奖励建模

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-15 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo2bd9a5022lslbakg74ei4z
- 原文链接：https://arxiv.org/abs/2604.13618

## AI 摘要

针对传统评分标准增强奖励建模依赖昂贵人工标注且易受低质标准误导的问题，本文提出C2（Cooperative yet Critical）框架。该框架通过让奖励模型与仅基于二元偏好训练的标准生成器进行批判性协作，合成有益与误导性标准对比样本训练生成器，并由验证器筛选有效标准。实验表明，C2在RM-Bench上提升6.5分，在AlpacaEval 2.0上提升长度控制胜率6.0分，使8B模型性能媲美使用4倍大模型生成标准的表现，实现了可扩展的可靠奖励建模。

## 正文

Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find that rubric generation is vulnerable to a failure of cooperation; low-quality rubrics actively mislead reward models rather than help. Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences. In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4times larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makes reward models more trustworthy in a scalable way.