论文评审中人工智能审稿人的局限与机遇:基于45位专家对《自然》系列论文的评审分析
阅读原文· arxiv.org本研究通过一项大规模专家标注实验,邀请物理、生物与健康科学领域的45位科学家,耗时469小时,对82篇《自然》系列论文的2960条批评意见(来自人类与AI评审)进行多维度评估。结果发现,由GPT-5.2驱动的AI评审代理在准确性、重要性与证据充分性的综合评分上,超过了每篇论文得分最高的人类评审员(60.0%对48.2%)。AI评审能发现26%人类未提及的独特问题,但其意见重叠度(21%)远高于人类(3%),并暴露出16种人类没有的反复性弱点,如子领域知识有限、多文件长上下文管理能力不足等。研究表明,当前AI评审员更适合作为人类评审的补充工具,而非完全替代。
With the advancement of AI capabilities, AI reviewers are beginning to be deployed in scientific peer review, yet their capability and credibility remain in question: many scientists simply view them as probabilistic systems without the expertise to evaluate research, while other researchers are more optimistic about their readiness without concrete evidence. Understanding what AI reviewers do well, where they fall short, and what challenges remain is essential. However, existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts (e.g., score alignment, acceptance prediction), which is insufficient to characterize their capabilities and limits. In this paper, we close this gap through a large-scale expert annotation study, in which 45 domain scientists in Physical, Biological, and Health Sciences spent 469 hours rating 2,960 individual criticisms (each targeting one specific aspect of a paper) from human-written and AI-generated reviews of 82 Nature-family papers on correctness, significance, and sufficiency of evidence. On a composite of all three dimensions, a reviewing agent powered by GPT-5.2 scores above each paper's top-rated human reviewer (60.0% vs. 48.2%, p = 0.009), while all three AI reviewers (including Gemini 3.0 Pro and Claude Opus 4.5) exceed the lowest-rated human across every dimension. AI reviewers' accurate criticisms are also more often rated significant and well-evidenced, and surface a distinct 26% of issues no human raises. However, AI reviewers overlap far more than humans do (21% vs. 3% for cross-reviewer pairs), and exhibit 16 recurring weaknesses humans do not share, such as limited subfield knowledge, lack of long context management over multiple files, and overly critical stance on minor issues. Overall, our results position current AI reviewers as complements to, not substitutes for, human reviewers.