组合式合成:通过原子分解与重组扩展代码 RLVR 训练规模
阅读原文· arxiv.org针对可验证奖励强化学习(RLVR)中足够有挑战性的代码任务稀缺、现有种子扩展法限制新颖性与难度的问题,提出原子分解与重组(ADR)框架。ADR 将代码任务分解为原子元素并受控重组,从而生成真正新颖且高难度的可验证代码任务。实验表明,ADR 在原创性、难度、多样性和测试质量上均优于现有基线,并在算法编程、工具使用和数据科学等多个下游领域的 RLVR 训练中持续带来更大的代码能力提升。
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as the cornerstone for shaping the remarkable coding abilities of Large Language Models (LLMs). However, the scalability of RLVR is severely constrained by the scarcity of sufficiently challenging verifiable code tasks that target near the model's edge of competence. Prior studies often rely on heuristic seed expansions for data synthesis, which severely limits both novelty and difficulty. Consequently, the training value of such data fails to scale proportionally with the size of its synthesis. To this end, we propose Atomic Decomposition and Recombination (ADR), a novel framework that generates verifiable code tasks via decomposition into atomic elements and controlled recombination, thereby enabling the generation of genuinely novel and challenging verifiable code tasks. Experiments and analysis demonstrate that ADR achieves superior originality, difficulty, diversity, and test quality over existing baselines, and consistently delivers greater improvements in code ability across RLVR in diverse downstream domains, including algorithmic programming, tool usage, and data science. Our work sheds light on a new paradigm for novel code task synthesis and scalable RLVR training.