PlanBench-XL:评估大规模工具生态中LLM智能体的长时域规划
阅读原文· arxiv.orgPlanBench-XL是一个交互式基准测试,包含327个零售任务和1,665个工具,用于评估LLM智能体在检索受限工具可见性下的长时域规划能力。智能体需迭代检索并调用工具以逐步逼近最终目标。可选阻塞机制通过缺失、失败或干扰的工具函数模拟不确定性,迫使智能体检测中断路径并运行时自适应。在10个领先LLM上的实验显示,GPT-5.4在无阻塞条件下准确率为51.90%,最严重阻塞下骤降至11.36%。分析表明,当故障缺乏明确错误信号或恢复需要更长替代工具路径时,智能体尤其脆弱。PlanBench-XL为诊断智能体规划失败提供了测试平台。
LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.