评估卡片:AI评估报告的可解读层
阅读原文· arxiv.orgAI评估结果规模庞大但报告不一致,导致读者难以跨来源比较、识别遗漏或追溯结论。Evaluation Cards通过整合基准元数据、评估运行数据和模型元数据,形成统一记录。方法包括:(1)从52篇论文和10次利益相关者访谈中推导报告模式;(2)实现四个可解释信号(可复现性、文档完整性、来源与风险、分数可比性),并针对研究与非研究受众提供不同读者模式;(3)部署监控工具,覆盖5816个模型、635个基准和101843个结果,揭示当前报告实践中的系统性缺口。
AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present , an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.