SCOPE:通过共同演化策略实现开放式任务的自我对弈
阅读原文· arxiv.orgSCOPE是一个无需数据、通过自我对弈提升模型开放式任务能力的框架。它通过共同演化两个策略来工作:一个“挑战者”生成基于文档的任务,一个“解答者”通过多轮检索来回答。模型自身的一个冻结副本作为自我裁判,为任务生成评分标准并打分。在三个7-8B参数的指令微调模型(Qwen2.5, Qwen3, OLMo-3)上进行的实验显示,SCOPE将开放式任务性能在八个基准上最高提升了10.4分,达到或超过了使用约9K精调提示训练的GRPO_data。尽管仅针对开放式任务训练,SCOPE在七个留出的简短问答基准上也带来了最高13.8分的提升。消融实验表明,共同演化挑战者对维持任务难度是必要的,检索与合成能力均有贡献,而生成质量是自我评判的瓶颈。
Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight benchmarks and matches or exceeds GRPO_data trained on ~9K curated prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.