Model Selection

The selection framework

Model selection should be driven by systematic evaluation, not brand preference or benchmark rankings. The framework:

1. Define requirements first: What task (classification, generation, reasoning, code)? What quality bar (human parity, 90% of GPT-4, etc.)? What latency budget (real-time <1s, interactive <5s, batch <60s)? What cost budget (per-query, per-month)?
2. Build a golden test set: 100–500 representative inputs with expected outputs or quality criteria. This is non-negotiable — you cannot select a model without a domain-specific eval.
3. Shortlist candidates: 3–5 models that plausibly meet your requirements. Include the obvious frontrunners and one or two cheaper alternatives.
4. Benchmark systematically: Run all candidates on your golden test set. Score automatically where possible, sample-human-review otherwise.
5. Calculate total cost of ownership: API cost + infrastructure + maintenance + model update costs.

Frontier API models

The current frontier model landscape (as of early 2025):

For most production use cases: start with GPT-4o-mini or Claude Haiku for high-volume/low-stakes tasks, GPT-4o or Claude 3.5 Sonnet for quality-sensitive tasks.

Open-weight self-hosted models

Self-hosted open models are cost-effective at high volume and necessary for data privacy requirements:

Break-even point for self-hosting: typically 1–5M tokens/day, where hosting cost (GPU cloud) becomes cheaper than API calls. Below this, API is almost always more economical when you include engineering overhead.

Running your own benchmarks

import openai
from anthropic import Anthropic
import json, asyncio

async def benchmark_models(test_cases: list, models: list) -> dict:
    results = {}
    for model_config in models:
        scores = []
        for tc in test_cases:
            # Run the model
            output = await call_model(model_config, tc["input"])
            # Score the output
            score = await llm_judge(tc["input"], output, tc.get("expected"))
            scores.append({"input": tc["input"], "output": output, "score": score})
        results[model_config["name"]] = {
            "scores": scores,
            "mean_score": sum(s["score"] for s in scores) / len(scores),
            "pass_rate": sum(1 for s in scores if s["score"] >= 0.7) / len(scores),
            "avg_latency_ms": ...,
            "avg_cost_usd": ...,
        }
    return results

# Sort by quality/cost ratio
ranked = sorted(results.items(), key=lambda x: x[1]["mean_score"] / x[1]["avg_cost_usd"], reverse=True)
for name, metrics in ranked:
    print(f"{name}: score={metrics['mean_score']:.2f}, cost=${metrics['avg_cost_usd']:.4f}")

Cost modelling

def monthly_cost_estimate(
    daily_queries: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    input_price_per_m: float,
    output_price_per_m: float,
) -> float:
    monthly_queries = daily_queries * 30
    input_cost = monthly_queries * avg_input_tokens / 1_000_000 * input_price_per_m
    output_cost = monthly_queries * avg_output_tokens / 1_000_000 * output_price_per_m
    return input_cost + output_cost

# Example: customer support bot
daily_queries = 10_000
avg_input = 1500   # system prompt + conversation history + user message
avg_output = 300   # typical response

print("GPT-4o:", monthly_cost_estimate(daily_queries, avg_input, avg_output, 2.50, 10.00))
print("GPT-4o-mini:", monthly_cost_estimate(daily_queries, avg_input, avg_output, 0.15, 0.60))
print("Claude Haiku:", monthly_cost_estimate(daily_queries, avg_input, avg_output, 0.25, 1.25))

# GPT-4o:     $12,900/month
# GPT-4o-mini:  $846/month  — if quality is acceptable, huge savings
# Claude Haiku: $1,275/month

Latency requirements

Latency is often the deciding factor that overrides quality considerations:

• Real-time (<500ms): Voice assistants, autocomplete, live search. Only fast API models (GPT-4o-mini, Gemini Flash) or self-hosted small models (7B, quantised). No 70B+ models.
• Interactive (<5s): Chat interfaces, form assistance. Most API models qualify. Consider streaming for perceived latency — first token in 500ms feels fast even if full response takes 4s.
• Batch (>10s acceptable): Document processing, overnight analysis. Use larger, higher-quality models. Batch APIs (OpenAI, Anthropic) give 50% cost reduction at 24h turnaround.

Streaming is critical for user-facing applications: stream=True in API calls means users see tokens as they're generated, dramatically improving perceived responsiveness even when total latency is unchanged.

Gotchas

Public benchmarks don't predict your task performance: MMLU, HumanEval, and MT-Bench measure specific capabilities. A model that ranks 5th on MMLU may rank 1st on your classification task. Always evaluate on your domain data.

Model versions change silently: OpenAI and Anthropic update models behind the same API endpoint. gpt-4o in March 2024 is not the same model as gpt-4o in December 2024. Pin to specific model versions (e.g., gpt-4o-2024-08-06) in production to avoid silent quality regressions.

Rate limits matter at scale: Enterprise tier rate limits are 10–100× higher than free/standard tiers. Factor in the lead time to upgrade limits — OpenAI rate limit increases can take days or weeks to approve.