GRPO、Dr. GRPO 与 DAPO:组标准差上的三种操作
阅读原文· arxiv.orgGRPO、Dr. GRPO 和 DAPO 三种推理训练方法看似不同,实则都调节同一个数值——组标准差。对于二值奖励(正确/错误),组标准差衡量同一提示下多个答案的不一致程度:答案对半开时最大,全一致时为零。GRPO 除以该标准差,Dr. GRPO 去掉除法,DAPO 丢弃标准差为零的组。论文证明三者是同一参数的不同设置,并给出组标准差同一性:不一致的组产生最大更新,一致的组更新为零。该结论在 Big-Math 难度数据集和受控训练中得到验证。
Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.