HealthAgentBench:面向挑战性前沿AI智能体的统一医疗专家基准套件
阅读原文· arxiv.orgHealthAgentBench发布,包含54项医疗任务(7个类别),每项任务模拟患者就诊全流程的端到端临床工作流。智能体需在极少指令下探索原始数据、操作复杂环境并执行多步解决方案。评估前沿智能体后,整体任务成功率低,最强且成本最优的Codex GPT-5.5仅达约42%成功率。Claude Code在医学影像任务上表现困难,而Codex GPT-5.5展现新兴能力。结合大搜索空间与组合推理需求的任务对当前所有智能体构成挑战。该基准套件已开源。
As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.