CEO-Bench:智能体能玩长期游戏吗?
阅读原文· arxiv.orgCEO-Bench通过模拟初创公司500天运营,评估AI智能体在不确定性中规划、获取信息、适应变化和协调多目标的能力。智能体需通过Python接口管理定价、营销、预算等决策,并处理嘈杂数据库。最强模型(Claude Opus 4.8、GPT-5.5)虽能编写复杂代码预测现金流、挖掘客户偏好,但仅勉强使余额维持起始的100万美元以上,无法持续盈利。该基准首次衡量驱动长期自适应进展所需的智能。
Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that simulates customer cohorts to forecast future cash and mines negotiation history to uncover hidden customer preferences. Even so, most state-of-the-art models struggle in this environment. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently turns a profit. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.