RODS:多轮工具智能体奖励驱动在线数据合成
阅读原文· arxiv.org多轮工具使用强化学习受限于静态数据集中信息样本快速耗尽。GRPO梯度集中在奖励方差最高的任务(Popoviciu上界所致),靠近智能体能力边界的样本贡献不成比例的梯度。RODS将进度奖励方差作为零成本边界检测器,持续识别边界样本,通过技能对齐重采样管道合成结构复杂度匹配的新变体,并维护与策略共同演化的动态回放缓冲区。从400个人工种子出发、维持约800样本的活动池,RODS性能堪比17K样本离线管道,所需轨迹约少20倍。
Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples near the agent's capability boundary -- where successes and failures are roughly balanced -- contribute disproportionately large policy gradients. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset. We propose RODS (Reward-driven Online Data Synthesis) to resolve this depletion. RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies such boundary samples, synthesizes new multi-turn variants matching their structural complexity (e.g., API topology and dependency depth) via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of ~800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories, and improves over fixed-data RL and environment augmentation in our controlled setting.