TUA-Bench:面向通用终端智能体的基准测试
阅读原文· arxiv.orgTUA-Bench是一个通用终端智能体基准测试,包含120个真实世界任务,覆盖文档编辑、邮件管理、实时网页信息搜索及科研与工程工作流五大类别。任务在真实终端中通过确定性脚本执行,采用基于执行的评分协议评估。最强前沿模型Claude Code搭配Claude Opus 4.8最大推理努力,整体性能65.8%,各任务类别间差距显著。该基准旨在推动从窄任务助手向多数字环境可靠运行的通用终端智能体转变。
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.