CocoaBench:统一数字智能体真实场景能力基准测试
阅读原文· arxiv.orgCocoaBench 基准测试针对统一数字智能体发布,通过人工设计的长程任务评估其灵活组合视觉、搜索与编程能力的实战表现。该基准采用自动评估函数确保跨架构评测的可靠性,同步推出的 CocoaAgent 提供轻量级脚手架以实现模型间的公平对比。测试结果显示,当前最优系统成功率仅为 45.1%,表明现有智能体在推理规划、工具使用及视觉定位等关键环节仍有显著提升空间。
LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.