REVES:基于修订与验证的测试时扩展训练框架
REVES是一种两阶段迭代训练框架,通过将成功恢复轨迹中的中间步骤转化为独立的修订与验证提示,聚焦于答案变换与错误识别,相比标准多轮强化学习降低了长程采样的计算开销。在LiveCodeBench上,使用公开测试用例作为反馈,较RL基线提升+6.5点,较标准多轮训练提升+4.0点。在circle packing任务上,仅用4B参数基础模型即匹配此前报告的SOTA结果。该方法还泛化到n_queens、mini_sudoku等分布外约束满足问题。代码已开源。
Test-time scaling via sequential revision has emerged as a powerful paradigm for enhancing Large Language Model (LLM) reasoning. However, standard post-training methods primarily optimize single-shot objectives, creating a fundamental misalignment with multi-step inference dynamics. While recent work treats this as multi-turn reinforcement learning (RL), conventional approaches optimize over the multi-step trajectories directly, failing to further exploit the high-quality mistakes in intermediate steps that model can learn from correcting them. We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization. By converting the intermediate steps (``near-miss'' answers) in the successful recovery trajectories into decoupled revision and verification prompts, our approach concentrates training on both effective answer transformation and error identification. This approach enables efficient off-policy data generation and reduces the computational overhead of long-horizon sampling compared to standard multi-turn RL. On LiveCodeBench, using publicly available test cases as feedback, we observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training. Beyond coding, our approach matches the previously reported SOTA result on circle packing while using the smallest base model (4B) and far fewer rollouts than the much larger evolutionary search systems. Math results under ground-truth verification further confirm improved correction ability. It also generalizes to out-of-distribution constraint-satisfaction puzzles such as n\_queens and mini\_sudoku, where correctness is defined entirely by problem constraints. Code is available at https://github.com/yxliu02/REVES.git.