个人相机胶卷视觉问答AI智能体(Camroll-Agent)
阅读原文· arxiv.org研究个人相机胶卷视觉问答场景,AI助手可访问用户相机胶卷并检索相关照片回答事实性或开放性问题。构建camroll数据集,包含50名用户、31,476张图像和2,500个问答对。设计camroll-agent对话式智能体,配备层次化记忆和最小工具集以高效导航大规模个性化视觉记忆。实验表明其优于多种基线方法,揭示个性化视觉记忆需要不同于标准长上下文文本记忆的方法,尤其在一致性、视觉细节和用户特定上下文方面。
We study the personal camera roll visual question answering setting. In this setting, a conversational AI assistant can access a user's personal camera roll and retrieve relevant photos to answer queries, ranging from simple factual questions (e.g., `Name of the food I tried yesterday?'') to more open-ended ones (e.g., `Recommend some dishes I have never eaten before''). Given the vast nature of the personal camera roll (i.e., multiple years, hundreds to thousands of photos), a successful AI assistant needs to understand a long-horizon, highly personalized visual content stream in order to navigate and locate the correct and/or relevant information. To support this, we collect and manually annotate questions that mimic real-world usage. The final dataset, camroll, contains 50 users, 31,476 images, and 2,500 QA pairs. We further design camroll-agent, a conversational AI agent equipped with hierarchical memory and a minimal set of tools for efficient navigation over large, personalized visual memory. Experimental results show that camroll-agent outperforms numerous baselines and methods for long-context understanding AI agents system. Together, the camroll dataset and camroll-agent highlight the gap in AI agents' long-context reasoning: personalized visual memory requires different approaches from standard long-context textual memory, especially when consistency, visual details, and user-specific context are present.