As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning 13 functional domains and 212 realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: (1)Sub-agent Collaboration---decomposing a complex task and delegating specialized work to capable sub-agents; (2)Memory Usage---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and (3)~Skill Usage---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.
MobilePA-Bench:面向复杂真实任务的移动端规划智能体评测基准
AI 导读
MobilePA-Bench 是一个交互式、有状态且以工具为中心的评测基准,用于评估移动端规划智能体的工具调用与规划能力。它运行在可执行沙箱中,覆盖 13 个功能域和 212 个真实移动工具,并额外评测子智能体协作、记忆使用和技能调用三个高级维度。实验显示,当前前沿 LLM 在严格工具排序、权限限制和意外运行时错误下性能显著下降。
HuggingFace Daily Papers(社区热门论文)
52
AI 编辑部评分,满分 100MobilePA-Bench:面向复杂真实任务的移动端规划智能体评测基准
MobilePA-Bench 是一个交互式、有状态且以工具为中心的评测基准,用于评估移动端规划智能体的工具调用与规划能力。它运行在可执行沙箱中,覆盖 13 个功能域和 212 个真实移动工具,并额外评测子智能体协作、记忆使用和技能调用三个高级维度。实验显示,当前前沿 LLM 在严格工具排序、权限限制和意外运行时错误下性能显著下降。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org