# RankJudge：一个用于多轮对话中大语言模型评判者的合成基准测试生成器

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-20 08:00
- AIHOT 分数：53
- AIHOT 链接：https://aihot.virxact.com/items/cmpmublz40rxfsl019i6ulk7f
- 原文链接：https://arxiv.org/abs/2605.21748

## AI 摘要

RankJudge是一个评估大语言模型作为评判者在基于参考文档的多轮对话中表现的基准测试生成器。它通过生成配对对话来工作，其中一个对话的单一轮次被注入缺陷，从而明确评判结果并精准定位错误类别。该基准在机器学习、生物医学和金融三个领域实施，对21个前沿大语言模型评判者进行了评估，并使用Bradley-Terry模型对其排名。RankJudge还能为对话对生成难度评分，用于动态筛选评估数据以降低标签噪音。

## 正文

As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. For simpler systems, human evaluation may be practical, but in complicated systems like conversational chatbots, the amount of generated text can overwhelm human annotation resources. Model developers have begun to rely heavily on auto-evaluation, where LLMs are also used to judge generation quality. However, existing LLM-as-a-judge benchmarks largely focus on simple Q\&A tasks that do not match the complexity of multi-turn conversations. We introduce RankJudge, a benchmark generator for evaluating LLM-as-a-judge on multi-turn conversations grounded in reference documents. RankJudge creates pairs of conversations where one conversation has a single flaw injected into one turn. This construction allows paired conversations to be labeled unambiguously as better or worse, and precisely isolates failure categories to individual turns, enabling a strict joint correctness criterion for judging. We implement RankJudge across the domains of machine learning, biomedicine, and finance, evaluate 21 frontier LLM judges, and rank those judges via the Bradley-Terry model. Our formulation also allows ranking each conversation pair with difficulty ratings, which we use to dynamically curate the evaluation slice to reduce label noise, as confirmed via human annotation. We find that judge rankings are stable under partial observability, coarser correctness criteria, and an alternative random-walk rating algorithm.