# PRECISE：基于预测驱动推理的统计可靠LLM排序评估方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-03 08:00
- AIHOT 分数：45
- AIHOT 链接：https://aihot.virxact.com/items/cmqfjnygc00juslq89kcirr5x
- 原文链接：https://arxiv.org/abs/2606.05308

## AI 摘要

PRECISE扩展预测驱动推理（PPI），结合少量人工标注与大量LLM评判，得到偏差校正的排序评估指标。PPI在任意LLM评判误差分布下均无偏。针对Precision@K等分层指标，将输出空间计算复杂度从O(2^|C|)降至O(2^K)。在ESCI基准上，30条人工标注加上Claude 3 Sonnet评判使Precision@4估计的标准误差从4.45降至3.50（降低21%）。生产系统中，该框架从100条标签和2小时领域专家标注中正确识别出三个系统变体的最优者，A/B测试确认该排名，日销售额提升407 bps。

## 正文

With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.