Chatbot Arena

How Chatbot Arena works

Chatbot Arena (LMSYS, launched May 2023) is a human preference evaluation platform. Users submit prompts to a chat interface that simultaneously queries two anonymous models. After seeing both responses, users click "Model A is better", "Model B is better", "Tie", or "Both are bad". Only after voting are the model identities revealed. This blind setup prevents model popularity bias from influencing ratings. Votes are aggregated via a Bradley-Terry statistical model to produce Elo-style rankings.

Bradley-Terry Elo model

The Arena uses the Bradley-Terry model (a generalization of Elo) to estimate each model's skill from pairwise comparisons. Model i's strength is θ_i; the probability that model i beats model j is P(i > j) = 1 / (1 + exp(θ_j - θ_i)). Given all pairwise vote outcomes, maximum likelihood estimation finds the θ values that best explain the observed comparisons. The resulting scores are converted to Elo scores (e.g. baseline at 1000) for interpretability.

import numpy as np
from scipy.optimize import minimize

def fit_bradley_terry(outcomes: list[tuple[int, int, float]]) -> np.ndarray:
    # outcomes: list of (model_i, model_j, fraction_i_wins)
    n_models = max(max(i, j) for i, j, _ in outcomes) + 1
    theta = np.zeros(n_models)

    def neg_log_likelihood(theta):
        nll = 0
        for i, j, frac in outcomes:
            p_i_wins = 1 / (1 + np.exp(theta[j] - theta[i]))
            nll -= frac * np.log(p_i_wins + 1e-10) + (1 - frac) * np.log(1 - p_i_wins + 1e-10)
        return nll

    result = minimize(neg_log_likelihood, theta, method="L-BFGS-B")
    scores = result.x
    # Normalise: set mean to 0, scale to Elo (400 per 10× odds)
    scores -= scores.mean()
    return scores * (400 / np.log(10))

What Arena measures

Arena rankings capture real human preference in open-ended chat — the closest proxy to "which model do users actually prefer". This is distinct from:

• MMLU / academic benchmarks: Measure knowledge recall, not chat quality
• MT-Bench: GPT-4 as judge has its own biases; 80 questions is a small sample
• Safety benchmarks: Arena doesn't specifically test refusals or harmful outputs

Arena correlates well with deployment satisfaction for consumer chat use cases. It's less useful for specialized domains (coding, science) unless filtered to domain-specific prompts.

Accessing Arena data

from datasets import load_dataset

# LMSYS releases anonymized vote data
dataset = load_dataset("lmsys/chatbot_arena_conversations")
print(dataset["train"][0].keys())
# ['question_id', 'model_a', 'model_b', 'winner', 'judge',
#  'conversation_a', 'conversation_b', 'turn', 'language', 'tstamp']

# Filter to English single-turn conversations
english_single = dataset["train"].filter(
    lambda x: x["language"] == "English" and x["turn"] == 1
)
print(f"{len(english_single)} conversations")

# Analyse win rates
win_counts = {}
for row in english_single:
    winner = row["winner"]  # "model_a", "model_b", "tie", "tie (bothbad)"
    model = row[winner] if winner in ("model_a", "model_b") else None
    if model:
        win_counts[model] = win_counts.get(model, 0) + 1

for model, wins in sorted(win_counts.items(), key=lambda x: -x[1])[:5]:
    print(f"{model}: {wins} wins")

Building your own arena

import openai, anthropic, random

def get_model_response(model_id: str, prompt: str) -> str:
    if "gpt" in model_id:
        client = openai.OpenAI()
        resp = client.chat.completions.create(
            model=model_id,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=1024,
        )
        return resp.choices[0].message.content
    elif "claude" in model_id:
        client = anthropic.Anthropic()
        resp = client.messages.create(
            model=model_id,
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.content[0].text
    raise ValueError(f"Unknown model: {model_id}")

def arena_battle(prompt: str, models: list[str]) -> dict:
    model_a, model_b = random.sample(models, 2)
    resp_a = get_model_response(model_a, prompt)
    resp_b = get_model_response(model_b, prompt)
    # Blind presentation — randomise which is shown first
    return {
        "prompt": prompt,
        "responses": [resp_a, resp_b],
        "model_ids": [model_a, model_b],  # hidden from rater
    }

Limitations

• Popularity bias despite blinding: Users who know model response styles may identify models even when anonymous. GPT-4-style verbose structured responses are recognisable.
• Prompt distribution: Arena prompts skew toward casual conversation and English. Code-heavy or technical prompts are underrepresented. Category-filtered leaderboards (Arena-Hard, coding-specific) address this partially.
• Evaluator quality: Anyone can vote. Non-expert raters may prefer confidently-wrong responses over hedged-but-accurate ones. Safety is not rewarded — models that refuse more often tend to rank lower.

Gotchas

• Elo confidence intervals: Models with few battles have wide confidence intervals. A model ranked #5 may be statistically indistinguishable from #3–#10. Look at bootstrap confidence intervals, not just point estimates.
• Arena-Hard vs Arena: Arena-Hard uses a fixed set of 500 challenging prompts with GPT-4 as judge, specifically to reduce noise. It correlates well with full Arena rankings and is reproducible. Use Arena-Hard for internal evaluations.
• Model updates: When OpenAI or Anthropic release updated model versions, the Arena entry may not clearly distinguish versions. "GPT-4" in early Arena data ≠ "GPT-4o" — check the model version strings carefully.