Last Translation Benchmark 发布,用人工评审的多模态样本测试机器翻译模型极限

HuggingFace Daily Papers(社区热门论文)·2026-09-03 08:00·1天前
AI 导读

研究者推出 Last Translation Benchmark,收集经同行评审的人工创作样本(文本、图像、音频、视频),用于打破主流机器翻译模型。每个样本附带手工编写的验证规则,描述具体失败情形,使评估可靠且可操作;该数据集为持续接收投稿的 live dataset,当前版本 LTBv1 收录 2026 年 9 月 1 日前被接受的贡献。

HuggingFace Daily Papers(社区热门论文)
39AI 编辑部评分,满分 100

Last Translation Benchmark 发布,用人工评审的多模态样本测试机器翻译模型极限

2026-09-03 08:00· 1天前
AI 导读

研究者推出 Last Translation Benchmark,收集经同行评审的人工创作样本(文本、图像、音频、视频),用于打破主流机器翻译模型。每个样本附带手工编写的验证规则,描述具体失败情形,使评估可靠且可操作;该数据集为持续接收投稿的 live dataset,当前版本 LTBv1 收录 2026 年 9 月 1 日前被接受的贡献。

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org