# LMSYS 与 Kaggle 联合举办人类偏好预测竞赛，奖金 10 万美元

- 来源：LMSYS：Blog（Chatbot Arena 团队）
- 发布时间：2024-05-02 00:00
- AIHOT 链接：https://aihot.virxact.com/items/cmnxbjke60080sln0wgvib9ss
- 原文链接：https://www.lmsys.org/blog/2024-05-02-kaggle-competition

## AI 摘要

LMSYS 与 Kaggle 联合发起一项人类偏好预测竞赛，总奖金池达 10 万美元。参赛者需构建预测模型，判断用户在大型语言模型（LLM）两两对决中更偏好哪个回答。竞赛基于 LMSYS Arena 的真实对战数据，旨在通过众包方式探索更准确的 LLM 评估方法，推动模型与人类偏好对齐。比赛面向全球开发者开放，获胜方案有望改进现有大模型排名机制。

## 正文

Contents

Overview

Background

Competition Details

LMSYS Kaggle Competition – Predicting Human Preference with $100,000 in Prizes

Overview

LMSYS and Kaggle are launching a human preference prediction competition! You are challenged to predict which responses users will prefer in head-to-head battles between Large Language Models (LLMs). You'll work with a dataset from the Chatbot Arena, containing conversations and user preferences across various LLMs. By developing a model that accurately predicts human preferences, you'll contribute to improving chatbot performance and alignment with user expectations. The training dataset includes over 55,000 real-world user and LLM conversations and user preferences, with personally identifiable information removed. Your solution submission will be tested on a hidden test set of 25,000 samples. The dataset includes real-world conversations with over 70 state-of-the-art LLMs, such as GPT-4, Claude 2, Llama 2, Gemini, and Mistral models. Click here to join the competition and download the dataset!

Background

Current LLM benchmarks often fail to capture real-world LLM usage, resulting in a discrepancy between model performance and user satisfaction. Platforms like Chatbot Arena allow users to submit questions and vote on preferred responses; however, the potential of this data has been largely untapped in developing models that predict and optimize for user preferences at scale. Predicting user preferences is essential for creating human-aligned conversational AI that delivers a satisfying user experience. Successful models could enable language models to dynamically adapt their output based on individual preferences across different contexts and use cases. Moreover, this competition aims to uncover the factors that drive user preferences beyond objective correctness. Many user questions are open-ended, and we have already found a correlation between user preference and subjective qualities like conversationality. This could also be one of the best testbeds for reward modeling in your RLHF algorithms.

Competition Details

The competition will run until August 5th, with a total prize of $100,000, featuring a $25,000 prize for 1st place, $20,000 prizes for 2nd through 4th places, and a $15,000 prize for 5th place. This is your opportunity to contribute to the advancement of human-aligned language models while gaining valuable insights into human preferences and decision-making. These insights could provide value to both the computer science and psychology communities, shedding light on the factors that shape human preferences in conversational AI.

$100,000

$25,000

$20,000

$15,000