OpenBioRQ:未解决生物医学研究问题的智能体基准测试
阅读原文· arxiv.orgOpenBioRQ 是一个包含 12,553 个未解决生物医学研究问题的智能体基准测试,覆盖 12 个领域。问题无固定答案密钥,迫使模型通过多步工具调用自主验证证据,从而评估其真实性及弃权能力。当前智能体极少捏造引用(超 99% 可解析),但约 15.9% 的链接指向无关论文。难度锚定在三个开源模型都无法回答的问题上;在最难子集中同系列模型仅解决约 17%,而前沿智能体(Gemini-3-Pro、Opus-4.7、GPT-5.5)表现跨度达 29-60%。困难问题中出现“智能体崩溃”——模型停止使用工具。引入冻结的每问题检查表后,评分者间一致性从 Spearman 0.35 提升至 0.82。
A working citation looks like proof -- but the fact that a link resolves does not mean the cited paper supports the claim. I find that current agentic models rarely fabricate citations (over 99% resolve), yet roughly 15.9% link to the wrong paper. Existing benchmarks miss this failure mode: when a question has a fixed answer key, a model can reproduce the expected source from that key rather than independently verifying that the source supports the claim. I introduce \openbiorq{}, a retrieval-grounded agentic benchmark of 12{,}553 unsolved biomedical research questions across 12 domains that treats open questions as a faithfulness-and-abstention probe. To my knowledge, this is the first biomedical benchmark to combine an agentic setting -- where the model must issue multiple tool calls -- with unsolved questions that have no answer key. Openness is verified against real follow-up evidence rather than a model's parametric knowledge. Difficulty is empirical: I anchor it on questions that three open-weight reference models fail to answer, rather than on subjective hardness labels. On this hardest subset, held-out models from the same lineage as the difficulty anchors solve only ~17%, while three independent frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) span a wide 29-60% range. The benchmark is thus hard, non-saturating (the best agent still leaves ~33-40\% unsolved), and discriminating across capability tiers. Beyond difficulty, I observe agentic collapse on the hardest questions, where agents stop using their tools. For the most collapse-prone model, blocking tool access entirely barely changes its score -- so tools stop paying off exactly where they are needed most. A frozen per-question checklist raises inter-judge agreement from Spearman 0.35 to 0.82.