LLM Evaluation in Practice

Why Evals Are Hard

Classic software testing is deterministic and binary: a function either returns the correct value or it doesn't. LLM evaluation is fundamentally different. Model outputs are non-deterministic, open-ended, and contextual. A response can be correct in multiple ways, and "correctness" itself is often subjective.

The eval trilemma defines the core tension: accuracy (does your metric actually measure what matters?), cost (can you run it frequently without bankrupt budget?), and speed (do you get feedback before your next deployment?). You can usually achieve only two.

No single eval covers all failure modes. You need a suite. Common mistakes: evaluating only on benchmark datasets (train-test contamination and narrow coverage), using BLEU/ROUGE for generation (poor correlation with human judgment), and ignoring regression testing over time. These shortcuts feel fast until production fails.

The Eval Funnel: Offline → Shadow → Live

Effective eval strategies have three layers. Each layer catches different failure modes and carries different risk profiles. Together they form a funnel that progressively de-risks model deployment.

The Three Layers

Layer 1 — Offline evals: Run before any deployment. Automated, fast, cheap. Catch obvious regressions on your golden set. Layer 2 — Shadow evals: New model handles real traffic but responses aren't shown to users. Compare outputs between old and new. Layer 3 — Online evals: A/B test. Real users, real stakes. Monitor engagement, satisfaction, thumbs-up/down signals.

import json, statistics, time
from openai import OpenAI
from enum import Enum

client = OpenAI()

class EvalStage(Enum):
    OFFLINE = "offline"   # run against golden set before deploy
    SHADOW  = "shadow"    # run new model in parallel, compare to prod
    LIVE    = "live"      # sample % of live traffic for quality check

def offline_eval(golden_path: str, model: str) -> dict:
    """Stage 1: evaluate against frozen golden set."""
    golden = [json.loads(l) for l in open(golden_path)]
    scores = []
    for item in golden:
        resp = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": item["input"]}],
            temperature=0.0
        ).choices[0].message.content
        # Exact-match or LLM-judge scoring
        score = 1.0 if item["expected"].lower() in resp.lower() else 0.0
        scores.append(score)
    return {"stage": "offline", "n": len(golden),
            "accuracy": round(statistics.mean(scores), 3),
            "passed": statistics.mean(scores) >= 0.80}

def shadow_eval(requests: list[dict], prod_model: str, new_model: str) -> dict:
    """Stage 2: run both models, flag where new model differs significantly."""
    diffs = []
    for req in requests[:100]:  # sample first 100
        prod_resp = client.chat.completions.create(
            model=prod_model,
            messages=[{"role": "user", "content": req["input"]}]
        ).choices[0].message.content
        new_resp = client.chat.completions.create(
            model=new_model,
            messages=[{"role": "user", "content": req["input"]}]
        ).choices[0].message.content
        # Simple diff: check response length ratio
        ratio = len(new_resp) / max(len(prod_resp), 1)
        if ratio < 0.5 or ratio > 2.0:
            diffs.append({"input": req["input"], "ratio": ratio})
    return {"stage": "shadow", "n": len(requests[:100]),
            "significant_diffs": len(diffs), "diff_rate": len(diffs)/100}

# Usage
result = offline_eval("golden_set.jsonl", "gpt-4o")
if result["passed"]:
    print(f"Offline eval PASSED ({result['accuracy']:.0%})")
else:
    raise SystemExit(f"Offline eval FAILED ({result['accuracy']:.0%})")

Metrics for Generation Quality

Exact match: For classification and extraction tasks. Not useful for open-ended generation. BLEU / ROUGE: n-gram overlap with a reference answer. Still used for translation and summarization but poorly correlates with actual quality. BERTScore: Embedding-based semantic similarity. Better than BLEU, still requires a reference. G-Eval: Use GPT-4 to score responses on rubric dimensions (fluency, coherence, relevance, accuracy). High correlation with human judgments.

The choice depends on your use case. For factual extraction, exact match is reliable. For creative generation, LLM judging is more trustworthy than n-gram overlap. Invest in building ground truth — a small set of human-evaluated examples that validates your automated metrics.

LLM-as-Judge

Use a capable LLM (GPT-4o, Claude) to evaluate outputs on custom rubrics. This scales human judgment without re-hiring annotators. But it introduces new failure modes: position bias (preferring the first response), verbosity bias (preferring longer answers), and self-serving bias (models prefer their own outputs).

Point-wise vs Pair-wise

Point-wise: Score each response independently on a 1–5 scale per dimension. Faster, easier to parallelize, but noisier. Pair-wise: Compare two responses, pick the better one. More reliable than point-wise because the relative comparison is clearer. Requires more judge calls but produces higher-quality signals.

Reference-free Evaluation

You don't need a ground-truth reference to use LLM-as-judge. The judge can evaluate coherence, safety, and helpfulness from first principles. This is more expensive than metrics but works for open-ended tasks where multiple correct answers exist.

Failure Modes and Mitigations

Position bias: Judge prefers first response. Mitigation: swap order and average. Verbosity bias: Longer answers score higher. Mitigation: require conciseness in the rubric. Self-serving bias: Judge model prefers outputs from its own family. Mitigation: use diverse judge models (GPT-4o, Claude, Llama).

Golden Sets and Regression Testing

Your golden dataset is a curated set of (input, expected output) pairs that define acceptable behavior. It's your product spec in eval form. Sources include real user queries (sampled and labeled), adversarial cases (red-team), and edge cases discovered in production.

Building and Growing Your Golden Set

Size guidance: Start with 100-200 cases for initial coverage. Grow to 500-1000 as your product matures. Prioritize diversity over raw size. Labeling: Have domain experts label these examples. A single label is better than disagreement. Versioning: Keep golden sets under version control. Track when cases were added and why.

Regression Testing Workflow

Run your golden set on every model change. Flag if score drops >2% on any category. Use slice analysis: don't just track overall accuracy — track by query type, language, user segment, and difficulty level. This catches regressions that hide in the aggregate.

Eval Frameworks and Infrastructure

Many frameworks exist to manage eval pipelines. Choose based on your team's preferences and your existing stack. The key is integrating evals into CI/CD so they block bad deployments automatically.

CI/CD Integration

Run offline evals on every PR. Block merge if score drops below threshold. Log every eval run with model version, prompt version, and dataset version — this enables trend analysis and bisecting regressions to specific changes. Use smaller judge models for fast CI runs (Llama 2, Mistral). Reserve GPT-4o for weekly deep evals.

Eval Anti-Patterns

Common pitfalls that lead teams to trust evals that don't actually measure product quality:

References

• Paper G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Liu et al. (2023). arXiv:2303.16634. — arxiv:2303.16634 ↗
• Paper Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. LMSYS (2024). arXiv:2306.05685. — arxiv:2306.05685 ↗
• Paper Judging LLM-as-Judge with an LLM-as-Judge. Dubois et al. (2024). arXiv:2310.05470. — arxiv:2310.05470 ↗

• Docs OpenAI Evals — GitHub. github.com/openai/evals ↗
• Docs DeepEval — GitHub. github.com/confident-ai/deepeval ↗
• Docs LangSmith — Eval Docs. docs.smith.langchain.com ↗
• Guide Braintrust — Eval Best Practices. braintrust.dev ↗

• Blog Liang et al. — HELM: Holistic Evaluation. stanford.edu/helm ↗
• Blog OpenAI. — Building Better Evals. openai.com/blog ↗