Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.
DRACO:用动态评分标准做长程智能体训练的细粒度信用分配
AI 导读
DRACO 提出在结果信号缺失的长程智能体训练中,动态生成评分标准并按轨迹打分,再把判断以闭式方法重分配到相关步骤,在 GRPO 中产生差异化逐步优势,不引入任何训练的归因模块。
HuggingFace Daily Papers(社区热门论文)
41
AI 编辑部评分,满分 100DRACO:用动态评分标准做长程智能体训练的细粒度信用分配
DRACO 提出在结果信号缺失的长程智能体训练中,动态生成评分标准并按轨迹打分,再把判断以闭式方法重分配到相关步骤,在 GRPO 中产生差异化逐步优势,不引入任何训练的归因模块。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org