PlanningBench:一个用于评估和训练大语言模型的可扩展、可验证规划数据生成框架
阅读原文· arxiv.org本文提出了PlanningBench框架,旨在为评估与训练大语言模型生成可扩展、多样且可验证的规划数据。框架源于真实场景,抽象出包含30余种任务类型、约束与难度因素的结构化分类体系,并基于此实现了约束驱动的数据合成流程,该流程具备自适应难度控制、质量过滤与实例级验证能力。研究利用该框架评估了前沿大模型,发现其在耦合约束下仍难以生成完整解决方案。此外,基于PlanningBench数据的强化学习不仅提升了模型在未知规划任务上的表现,也增强了其指令遵循能力,并表明明确的最优解能提供更稳定的训练信号。
Planning is a fundamental capability for large language models (LLMs) because such complex tasks require models to coordinate goals, constraints, resources, and long-term consequences into executable and verifiable solutions. Existing planning benchmarks, however, usually treat planning data as fixed collections of instances rather than controllable generation targets. This limits scenario coverage, ties difficulty to surface-level proxies rather than structural sources, and offers limited support for scalable generation, automatic verification, or planning-oriented training. We introduce PlanningBench, a framework for generating scalable, diverse, and verifiable planning data for both evaluation and training. PlanningBench starts from real planning scenarios and abstracts practical workflows into a structured taxonomy of more than 30 task types, subtasks, constraint families, and difficulty factors. Guided by this taxonomy, a constraint-driven synthesis pipeline instantiates self-contained planning problems with adaptive difficulty control, quality filtering, and instance-level verification checklists. This shifts planning data construction from fixed benchmark collection to controllable generation while preserving realistic task grounding. We use PlanningBench to evaluate open-source and closed-source frontier LLMs, and find that current models still struggle to produce complete solutions under coupled constraints. Beyond evaluation, reinforcement learning on verified PlanningBench data improves performance on unseen planning benchmarks and broader instruction-following tasks. Further analysis suggests that determinate or well-specified optimal solutions provide clearer reward signals and more stable training dynamics. Overall, PlanningBench provides a controllable source of planning data for diagnosing and improving generalizable planning abilities in LLMs.