GD2PO:通过分组动态奖励解耦策略优化缓解多奖励冲突
阅读原文· arxiv.org大语言模型后训练强化学习需同时优化多个可能冲突的奖励维度。现有GDPO方法将整体分数分解为独立奖励组分别计算损失,但单个rollout在不同维度上可能产生正负相反的advantage,导致信号抵消。受DAPO启发,GD²PO引入冲突感知过滤机制,屏蔽奖励维度间严重不一致的rollout,防止信号抵消,同时保留并增强有效advantage幅度,加速学习。还采用查询级重加权动态调整各查询更新强度。在工具调用、人类偏好对齐等多奖励场景实验中,GD²PO显著优于现有基线。代码已开源。
As LLMs advance, post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities. This shift demands new algorithms capable of optimizing diverse and potentially competing objectives simultaneously. To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group. However, this strategy still encounters multi-reward conflicts: a single rollout can yield positive advantages on certain reward dimensions but negative ones on others, causing opposing signals to cancel each other out during aggregation, further hindering RL training efficiency. Inspired by Dynamic sAmpling Policy Optimization (DAPO), which improves RL training efficiency by filtering out ineffective rollouts with near-zero advantages, we propose Group-Dynamic reward-Decoupled Policy Optimization (GD^2PO). Specifically, GD^2PO employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement. By preventing conflicting signals from canceling each other out, this masking strategy preserves and enhances the magnitude of effective RL advantages, thereby significantly accelerating learning efficiency. Furthermore, we introduce query-level reweighting to dynamically adjust the update intensity of each query based on its overall reward consensus. Experiments on various multi-reward scenarios, including tool calling and human preference alignment, demonstrate that GD^2PO consistently and significantly outperforms existing baselines. The code is available at https://github.com/Qwen-Applications/GD2PO.