VitaBench 2.0:评估长期用户交互中的个性化与主动智能体
阅读原文· arxiv.orgVitaBench 2.0是一个专注于评估大语言模型智能体在长期、碎片化用户交互中表现的基准。其任务按时间顺序组织,要求模型从异构交互中持续提取并更新用户偏好。基准通过设计需要主动向用户或环境获取缺失信息的任务来评估智能体的主动性,并提供了可扩展的内存接口。对前沿模型的评测显示,即使最先进的模型在现实个性化任务上仍面临重大挑战。分析揭示了当前智能体在实际个性化决策中的失败模式与能力瓶颈。
Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.