BenchEvolver:以解决方案为中心的进化式前沿任务合成
阅读原文· arxiv.org前沿大语言模型在LiveCodeBench上已饱和,易分题Pass@1超99%,平均超90%。新提出的BenchEvolver框架以解决方案为中心,通过结构化变换自动进化已有编码问题的参考解,再从进化后解推导题目与测试用例,从而可扩展构造更高质量、多样、困难且结果可验证的任务。应用于LiveCodeBench和SciCode后,进化任务难度显著提升。整理出的LiveCodeBench-Plus含91道题,前沿模型Pass@1仅27.5%–62.6%,恢复了强编码模型间的清晰区分。进化任务对生成模型自身也有挑战性,可用于自我提升。基于进化任务的强化学习在gpt-oss-20b上使LCB v6 Hard和LCB-Pro Easy的Pass@1分别提升+8.7和+8.3,超出仅用原始任务训练的增益70.7%和34.8%。
The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.