RACES:递归自动化环境组合以提升推理泛化
阅读原文· arxiv.orgRACES(Recursive Automated Composition for Environment Scaling)将可验证环境视为递归组装的构建块,当输出类型与输入类型匹配时自动融合为新环境。基于300个基础环境,定义SEQUENTIAL、PARALLEL、SORT、SELECT四种组合算子,诱导多样推理模式。在DeepSeek-R1-Distill-Qwen-14B上平均提升3.1分(从48.2到51.3),Qwen3-14B提升2.3分(从58.8到61.1),均在六个未见基准上测得。仅用50个基础环境即可达到300个环境的训练效果,环境利用效率显著。
Reinforcement Learning (RL) with verifiable environments has emerged as a powerful approach for enhancing the reasoning capabilities of Large Language Models (LLMs). While prior research demonstrates that scaling environment quantity improves RL performance, existing manual or individual construction methods suffer from linear scaling limits, thereby hindering scalable reasoning generalization. This paper introduces RACES (Recursive Automated Composition for Environment Scaling), a framework that conceptualizes verifiable environments as composable building blocks that can be recursively assembled. The key insight is that when the codomain (output type) of one environment matches the domain (input type) of another, they can be automatically fused into a new verifiable environment, enabling recursive composition. RACES is implemented with 300 individual environments and defines a set of composition operators (SEQUENTIAL, PARALLEL, SORT, and SELECT) that induce diverse reasoning patterns. Extensive experiments show that RL training on these composite environments consistently enhances reasoning generalization. Specifically, RACES improves DeepSeek-R1-Distill-Qwen-14B by an average of 3.1 points (from 48.2 to 51.3) and boosts Qwen3-14B performance from 58.8 to 61.1 on six benchmarks, which are unseen during the construction of training environments. Moreover, RACES achieves performance comparable to training on 300 individual environments using only 50 base environments, demonstrating significant efficiency in environment utilization.