Does style matter? Disentangling style and substance in Chatbot Arena
Why is GPT-4o-mini so good? Why does Claude rank so low, when anecdotal experience suggests otherwise?
We have answers for you. We controlled for the effect of length and markdown, and indeed, the ranking changed. This is just a first step towards our larger goal of disentangling substance and style in Chatbot Arena leaderboard.
Check out the results below! Style indeed has a strong effect on models’ performance in the leaderboard. This makes sense—from the perspective of human preference, it’s not just what you say, but how you say it. But now, we have a way of separating the effect of writing style from the content, so you can see both effects individually.
When controlling for length and style, we found noticeable shifts in the ranking. GPT-4o-mini and Grok-2-mini drop below most frontier models, and Claude 3.5 Sonnet, Opus, and Llama-3.1-405B rise substantially. In the Hard Prompt subset, Claude 3.5 Sonnet ties for #1 with chatgpt-4o-latest and Llama-3.1-405B climbs to #3. We are looking forward to seeing what the community does with this new tool for disaggregating style and substance!
Overall Ranking + Style Control
Figure 1. Overall Chatbot Arena ranking vs Overall Chatbot Arena ranking where answer length, markdown header count, markdown bold count, and markdown list element count are being “controlled”.
Does style matter? Disentangling style and substance in Chatbot Arena
Why is GPT-4o-mini so good? Why does Claude rank so low, when anecdotal experience suggests otherwise?
We have answers for you. We controlled for the effect of length and markdown, and indeed, the ranking changed. This is just a first step towards our larger goal of disentangling substance and style in Chatbot Arena leaderboard.
Figure 2. Hard Prompt category ranking vs Hard Prompt category ranking where answer length, markdown header count, markdown bold count, and markdown list element count are being “controlled”.
Full Leaderboard with Style Control
Please find the below links to leaderboard and colab notebook. We will be rolling out style control soon to all the categories. Stay tuned!
Leaderboard link
Colab link
Methodology
High-Level Idea. The goal here is to understand the effect of style vs substance on the Arena Score. Consider models A and B. Model A is great at producing code, factual and unbiased answers, etc., but it outputs short and terse responses. Model B is not so great on substance (e.g., correctness), but it outputs great markdown, and gives long, detailed, flowery responses. Which is better, model A, or model B?
The answer is not one dimensional. Model A is better on substance, and Model B is better on style. Ideally, we would have a way of teasing apart this distinction: capturing how much of the model’s Arena Score is due to substance or style.
Our methodology is a first step towards this goal. We explicitly model style as an independent variable in our Bradley-Terry regression. For example, we added length as a feature—just like each model, the length difference has its own Arena Score! By doing this, we expect that the Arena Score of each model will reflect its strength, controlled for the effect of length.
Please read below for the technical details. We also controlled not just for length, but also a few other style features. As a first version, we propose controlling
Answer token length
Number of markdown headers
Number of markdown bold elements
Number of markdown lists
We publicly release our data with vote and style elements and code on google colab! You can try out experimenting with style control now. More improvements to come, and please reach out if you want to help contribute!
Background. To produce the results above, we controlled for the effect of style by adding extra “style features” into our Bradley-Terry regression. This is a standard technique in statistics, and has been recently used in LLM evaluations [1]. Additionally, there are studies suggesting potential bias for “pretty” and more detailed responses in humans [2, 3]. The idea is that, by including any confounding variables (e.g. response length) in the regression, we can attribute any increase in strength to the confounder, as opposed to the model. Then, the Bradley-Terry coefficient will be more reflective of the model’s intrinsic ability, as opposed to possible confounders. The definition of a confounder is to some extent up to our interpretation; as our style features, we use the (normalized) difference in response lengths, the number of markdown headers, and the number of lists.
More formally, consider vectors X1,…,Xn∈RMX_1, \ldots, X_n \in \mathbb{R}^MX1,…,Xn∈RM and Y1,…,Yn∈{0,1}Y_1, \ldots, Y_n \in \{0,1\}Y1,…,Yn∈{0,1}, where nnn is the number of battles and MMM is the number of models.
For every i∈[n]i \in [n]i∈[n], We have that Xi,m=1X_{i,m}=1Xi,m=1 only if model m∈[M]m \in [M]m∈[M] is the model shown in the left-hand side in Chatbot Arena, and Xi,m=−1X_{i,m}=-1Xi,m=−1 only if it is shown on the right. That is, XiX_iXi is a vector with two nonzero elements. The outcome YiY_iYi takes the value Yi=1Y_i=1Yi=1 if the left-hand model wins, and Yi=0Y_i=0Yi=0 otherwise.
The standard method for computing the Arena Score (i.e., the Bradley-Terry coefficients, which we formerly called the Elo score) is to run a logistic regression of YiY_iYi onto XiX_iXi. That is, for every model mmm, we associate a scalar β^m\hat{\beta}_mβ^m that describes its strength, and the vector β^\hat{\beta}β^ is determined by solving the following logistic regression:
where BCELoss\mathsf{BCELoss}BCELoss represents the binary cross-entropy loss. (In practice, we also reweight this objective to handle non-uniform model sampling, but let’s ignore that for now.)
Style Control
Now, for every battle i∈[n]i \in [n]i∈[n], let’s say that in addition to XiX_iXi that we observe some additional style features, Zi∈RSZ_i \in \mathbb{R}^SZi∈RS. These style features can be as simple or complicated as you want. For example, ZiZ_iZi could just be the difference in response lengths of the two models, in which case S=1S=1S=1. Or, we could have S>1S>1S>1 and include other style-related features, for example, the number of markdown headers, common words associated with refusal, or even style features that are automatically extracted by a model!
Here, we define each style feature as normalize (featureA−featureBfeatureA+featureB)\text{normalize }\left(\frac{\text{feature}_A - \text{feature}_B}{\text{feature}_A + \text{feature}_B}\right)normalize (featureA+featureBfeatureA−featureB)
For example, the first new feature, token length difference between answer A and answer B, would be expressed as normalize (lengthA−lengthBlengthA+lengthB)\text{normalize }\left(\frac{\text{length}_A - \text{length}_B}{\text{length}_A + \text{length}_B}\right)normalize (lengthA+lengthBlengthA−lengthB)
We divide the difference by the sum of both answers' token length to make the length difference proportional to the pairwise answer token lengths. An answer with 500 tokens is roughly equal in length to an answer with 520 tokens, while an answer with 20 tokens is very different from an answer with 40 tokens, even though the difference is 20 tokens for both scenarios. Alternatively, AlpacaEval LC uses the following normalization technique.
The idea of style control is very basic. We perform the same logistic regression as before, but with some extra, additive style coefficients: β^,γ^=argminβ∈RM,γ∈RS1n∑i=1nBCELoss(sigmoid(Xi⊤β+Zi⊤γ),Yi).\hat{\beta}, \hat{\gamma} = \arg \min_{\beta \in \mathbb{R}^M, \gamma \in \mathbb{R}^S} \frac{1}{n}\sum\limits_{i=1}^n \mathsf{BCELoss}(\mathsf{sigmoid}(X_i^\top \beta + Z_i^{\top}\gamma), Y_i).β^,γ^=argminβ∈RM,γ∈RSn1i=1∑nBCELoss(sigmoid(Xi⊤β+Zi⊤γ),Yi). We refer to the results β^\hat{\beta}β^ and γ^\hat{\gamma}γ^ as the “model coefficients” and the “style coefficients” respectively. The model coefficients have the same interpretation as before; however, they are controlled for the effect of style, which is explicitly modeled by the style coefficients!
When the style coefficients are big, that means that the style feature has a big effect on the response. To define “big”, you need to properly normalize the style coefficients so they can be compared. All in all, when analyzing the style coefficients, we found that length was the dominant style factor. All other markdown effects are second order.
We report the following coefficient for each style attribute across different methods of controlling the style.
Length Markdown List Markdown Header Markdown Bold Control Both 0.249 0.031 0.024 0.019 Control Markdown Only - 0.111 0.044 0.056 Control Length Only 0.267 - - -
Ablation
Next, we compare the ranking changes between controlling for answer length only, markdown element only, and both. We present the Chatbot Arena Overall table first.
We want to continue building a pipeline to disentangle style and substance in the arena. Although controlling for style is a big step forward, our analysis is still observational. There are possible unobserved confounders such as positive correlation between length and substantive quality that are not accounted for by our study. For example, well-known example of a possible unobserved confounder that might positively impact both length and quality is a chain-of-thought explanation for a reasoning question.
To address these limitations, we are looking forward to implementing causal inference in our pipeline, and running prospective randomized trials to assess the effect of length, markdown, and more. Our pipeline for style control will be changing as we continue to improve our system and refine the analysis. Stay tuned, and let us know if you want to help!
Reference
[1] Dubois et al. “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators”, arXiv preprint
[2] Chen et al. “Humans or LLMs as the Judge? A Study on Judgement Bias”, arXiv preprint
[3] Park et al. “Disentangling Length from Quality in Direct Preference Optimization”, arXiv preprint
Citation
@misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} }
@misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} }
Check out the results below! Style indeed has a strong effect on models’ performance in the leaderboard. This makes sense—from the perspective of human preference, it’s not just what you say, but how you say it. But now, we have a way of separating the effect of writing style from the content, so you can see both effects individually.
When controlling for length and style, we found noticeable shifts in the ranking. GPT-4o-mini and Grok-2-mini drop below most frontier models, and Claude 3.5 Sonnet, Opus, and Llama-3.1-405B rise substantially. In the Hard Prompt subset, Claude 3.5 Sonnet ties for #1 with chatgpt-4o-latest and Llama-3.1-405B climbs to #3. We are looking forward to seeing what the community does with this new tool for disaggregating style and substance!
Overall Ranking + Style Control
Figure 1. Overall Chatbot Arena ranking vs Overall Chatbot Arena ranking where answer length, markdown header count, markdown bold count, and markdown list element count are being “controlled”.
Hard Prompt Ranking + Style Control
Figure 2. Hard Prompt category ranking vs Hard Prompt category ranking where answer length, markdown header count, markdown bold count, and markdown list element count are being “controlled”.
Full Leaderboard with Style Control
Please find the below links to leaderboard and colab notebook. We will be rolling out style control soon to all the categories. Stay tuned!
Leaderboard link
Colab link
Methodology
High-Level Idea. The goal here is to understand the effect of style vs substance on the Arena Score. Consider models A and B. Model A is great at producing code, factual and unbiased answers, etc., but it outputs short and terse responses. Model B is not so great on substance (e.g., correctness), but it outputs great markdown, and gives long, detailed, flowery responses. Which is better, model A, or model B?
The answer is not one dimensional. Model A is better on substance, and Model B is better on style. Ideally, we would have a way of teasing apart this distinction: capturing how much of the model’s Arena Score is due to substance or style.
Our methodology is a first step towards this goal. We explicitly model style as an independent variable in our Bradley-Terry regression. For example, we added length as a feature—just like each model, the length difference has its own Arena Score! By doing this, we expect that the Arena Score of each model will reflect its strength, controlled for the effect of length.
Please read below for the technical details. We also controlled not just for length, but also a few other style features. As a first version, we propose controlling
Answer token length
Number of markdown headers
Number of markdown bold elements
Number of markdown lists
We publicly release our data with vote and style elements and code on google colab! You can try out experimenting with style control now. More improvements to come, and please reach out if you want to help contribute!
Background. To produce the results above, we controlled for the effect of style by adding extra “style features” into our Bradley-Terry regression. This is a standard technique in statistics, and has been recently used in LLM evaluations [1]. Additionally, there are studies suggesting potential bias for “pretty” and more detailed responses in humans [2, 3]. The idea is that, by including any confounding variables (e.g. response length) in the regression, we can attribute any increase in strength to the confounder, as opposed to the model. Then, the Bradley-Terry coefficient will be more reflective of the model’s intrinsic ability, as opposed to possible confounders. The definition of a confounder is to some extent up to our interpretation; as our style features, we use the (normalized) difference in response lengths, the number of markdown headers, and the number of lists.
More formally, consider vectors X1,…,Xn∈RMX_1, \ldots, X_n \in \mathbb{R}^MX1,…,Xn∈RM and Y1,…,Yn∈{0,1}Y_1, \ldots, Y_n \in \{0,1\}Y1,…,Yn∈{0,1}, where nnn is the number of battles and MMM is the number of models.
For every i∈[n]i \in [n]i∈[n], We have that Xi,m=1X_{i,m}=1Xi,m=1 only if model m∈[M]m \in [M]m∈[M] is the model shown in the left-hand side in Chatbot Arena, and Xi,m=−1X_{i,m}=-1Xi,m=−1 only if it is shown on the right. That is, XiX_iXi is a vector with two nonzero elements. The outcome YiY_iYi takes the value Yi=1Y_i=1Yi=1 if the left-hand model wins, and Yi=0Y_i=0Yi=0 otherwise.
The standard method for computing the Arena Score (i.e., the Bradley-Terry coefficients, which we formerly called the Elo score) is to run a logistic regression of YiY_iYi onto XiX_iXi. That is, for every model mmm, we associate a scalar β^m\hat{\beta}_mβ^m that describes its strength, and the vector β^\hat{\beta}β^ is determined by solving the following logistic regression:
where BCELoss\mathsf{BCELoss}BCELoss represents the binary cross-entropy loss. (In practice, we also reweight this objective to handle non-uniform model sampling, but let’s ignore that for now.)
Style Control
Now, for every battle i∈[n]i \in [n]i∈[n], let’s say that in addition to XiX_iXi that we observe some additional style features, Zi∈RSZ_i \in \mathbb{R}^SZi∈RS. These style features can be as simple or complicated as you want. For example, ZiZ_iZi could just be the difference in response lengths of the two models, in which case S=1S=1S=1. Or, we could have S>1S>1S>1 and include other style-related features, for example, the number of markdown headers, common words associated with refusal, or even style features that are automatically extracted by a model!
Here, we define each style feature as normalize (featureA−featureBfeatureA+featureB)\text{normalize }\left(\frac{\text{feature}_A - \text{feature}_B}{\text{feature}_A + \text{feature}_B}\right)normalize (featureA+featureBfeatureA−featureB)
For example, the first new feature, token length difference between answer A and answer B, would be expressed as normalize (lengthA−lengthBlengthA+lengthB)\text{normalize }\left(\frac{\text{length}_A - \text{length}_B}{\text{length}_A + \text{length}_B}\right)normalize (lengthA+lengthBlengthA−lengthB)
We divide the difference by the sum of both answers' token length to make the length difference proportional to the pairwise answer token lengths. An answer with 500 tokens is roughly equal in length to an answer with 520 tokens, while an answer with 20 tokens is very different from an answer with 40 tokens, even though the difference is 20 tokens for both scenarios. Alternatively, AlpacaEval LC uses the following normalization technique.
The idea of style control is very basic. We perform the same logistic regression as before, but with some extra, additive style coefficients: β^,γ^=argminβ∈RM,γ∈RS1n∑i=1nBCELoss(sigmoid(Xi⊤β+Zi⊤γ),Yi).\hat{\beta}, \hat{\gamma} = \arg \min_{\beta \in \mathbb{R}^M, \gamma \in \mathbb{R}^S} \frac{1}{n}\sum\limits_{i=1}^n \mathsf{BCELoss}(\mathsf{sigmoid}(X_i^\top \beta + Z_i^{\top}\gamma), Y_i).β^,γ^=argminβ∈RM,γ∈RSn1i=1∑nBCELoss(sigmoid(Xi⊤β+Zi⊤γ),Yi). We refer to the results β^\hat{\beta}β^ and γ^\hat{\gamma}γ^ as the “model coefficients” and the “style coefficients” respectively. The model coefficients have the same interpretation as before; however, they are controlled for the effect of style, which is explicitly modeled by the style coefficients!
When the style coefficients are big, that means that the style feature has a big effect on the response. To define “big”, you need to properly normalize the style coefficients so they can be compared. All in all, when analyzing the style coefficients, we found that length was the dominant style factor. All other markdown effects are second order.
We report the following coefficient for each style attribute across different methods of controlling the style.
Length Markdown List Markdown Header Markdown Bold Control Both 0.249 0.031 0.024 0.019 Control Markdown Only - 0.111 0.044 0.056 Control Length Only 0.267 - - -
Ablation
Next, we compare the ranking changes between controlling for answer length only, markdown element only, and both. We present the Chatbot Arena Overall table first.
We want to continue building a pipeline to disentangle style and substance in the arena. Although controlling for style is a big step forward, our analysis is still observational. There are possible unobserved confounders such as positive correlation between length and substantive quality that are not accounted for by our study. For example, well-known example of a possible unobserved confounder that might positively impact both length and quality is a chain-of-thought explanation for a reasoning question.
To address these limitations, we are looking forward to implementing causal inference in our pipeline, and running prospective randomized trials to assess the effect of length, markdown, and more. Our pipeline for style control will be changing as we continue to improve our system and refine the analysis. Stay tuned, and let us know if you want to help!
Reference
[1] Dubois et al. “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators”, arXiv preprint
[2] Chen et al. “Humans or LLMs as the Judge? A Study on Judgement Bias”, arXiv preprint
[3] Park et al. “Disentangling Length from Quality in Direct Preference Optimization”, arXiv preprint
Citation
@misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} }
@misc{chiang2024chatbot, title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}, author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica}, year={2024}, eprint={2403.04132}, archivePrefix={arXiv}, primaryClass={cs.AI} }