GenAI in Production

Why GenAI Production Is Uniquely Hard

Running LLMs in production is fundamentally different from running traditional software. Traditional software is deterministic (same input → same output) and fast. LLMs are probabilistic, slow, and expensive. This changes everything about infrastructure, monitoring, and reliability.

Latency

LLMs are slow. Generating one token takes ~10-100ms on modern hardware. A 100-token response takes 1-10 seconds. Users expect responses in under 1-3 seconds. Solution: streaming (show tokens as they arrive), smaller models, and inference optimization (quantization, distillation, KV caching).

Cost

LLM inference is expensive. Each token costs money (OpenAI: $0.001-0.01 per 1K tokens). A typical conversation costs $0.10-1.00. Multiply by millions of users and costs spiral. Solution: prompt caching, smaller models, local inference, aggressive optimization.

Non-Determinism

Same prompt → different output. Temperature, sampling strategy, model version all affect results. Traditional software assumes determinism; production LLM systems need to handle randomness. Solutions: temperature control, output parsing, fallback models, human review loops.

Vendor Dependency

Using API vendors (OpenAI, Anthropic) means your app depends on their availability and pricing. They can change prices, deprecate models, or degrade performance. Solutions: multi-provider routing, fallbacks, caching, budget alerts.

The Three Pillars of Production GenAI

Production GenAI systems rest on three pillars. All three must be strong, or the system fails.

Pillar 1: Infrastructure & Serving

What: How to run models (cloud GPUs, inference optimizers, caching layers). Key questions: Where do we run models? How do we scale? How do we cache? How do we handle failures? Tools: vLLM, Ollama, Together AI, Baseten, Replicate.

Pillar 2: System Design Patterns

What: How to architect a system that uses LLMs reliably. Key questions: How do we route requests? How do we degrade gracefully? How do we fall back? How do we make LLM outputs deterministic? Patterns: Multi-provider routing, output caching, prompt versioning, circuit breakers.

Pillar 3: Production Engineering

What: How to monitor, debug, and operate LLM systems. Key questions: What metrics matter? How do we detect failures? How do we debug why an LLM failed? How do we track costs? Tools: Logging, tracing, dashboards, cost monitoring, user feedback loops.

Pillar 1: Infrastructure & Model Serving

Where and how you run models affects everything: latency, cost, control, and reliability.

Hosted APIs (OpenAI, Anthropic, Mistral)

Pros: No infrastructure to manage, always updated, highest quality models. Cons: No control, vendor lock-in, expensive, rate limits, data privacy. Best for: Prototypes, startups, apps where control isn't critical.

Self-Hosted (vLLM, Ollama, llama.cpp)

Pros: Full control, cheaper at scale, data stays local, can customize. Cons: Manage infrastructure, latency optimization, model updates on you. Best for: Production apps, on-prem deployments, privacy-critical work.

Inference Optimization Techniques

Quantization: Reduce model size and latency (int8 instead of fp32, 10x faster, minimal quality loss). Distillation: Train a small model to mimic a large one (smaller, faster, cheaper). Batching: Process multiple requests together (higher throughput). KV Caching: Cache attention keys/values (large speedup for generation).

Multi-Provider Strategy

Don't depend on one vendor. Use a router that tries primary provider (e.g., GPT-4), falls back to secondary (e.g., Claude), then tertiary (e.g., local model). Handles outages, rate limits, and cost optimization.

import hashlib, time, json
from functools import lru_cache
from openai import OpenAI

client = OpenAI()
_cache: dict = {}   # replace with Redis in production

def semantic_cache_key(prompt: str, model: str) -> str:
    """Exact-match cache key (use embedding similarity for semantic caching)."""
    return hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()

def tracked_completion(prompt: str, model: str = "gpt-4o",
                       use_cache: bool = True) -> dict:
    """LLM call with timing, caching, and basic observability."""
    key = semantic_cache_key(prompt, model)

    if use_cache and key in _cache:
        return {"response": _cache[key], "source": "cache", "latency_ms": 0}

    start = time.perf_counter()
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    latency_ms = (time.perf_counter() - start) * 1000
    content = resp.choices[0].message.content

    _cache[key] = content
    return {
        "response": content,
        "source": "llm",
        "latency_ms": round(latency_ms),
        "tokens": resp.usage.total_tokens,
        "model": model
    }

result = tracked_completion("What is the KV cache in LLMs?")
print(f"[{result['source']}] {result['latency_ms']}ms: {result['response'][:80]}...")

Pillar 2: System Design Patterns

How to build systems that use LLMs reliably at scale.

Output Caching & Deduplication

Many users ask similar questions. Cache outputs (question → answer) with expiration. Before calling the LLM, check cache. Can reduce LLM calls by 50-80% depending on use case. Use Redis or similar.

Prompt Versioning

Track prompt changes. If you improve a prompt, increment version. Old requests use old prompt, new requests use new prompt. Enables A/B testing and debugging ("this user is on prompt v2, not v3").

Output Guardrails

LLMs generate text freely; sometimes that's wrong (harmful, off-topic, wrong format). Add a filter layer: parse output, validate against schema, check for sensitive content, detect hallucinations. Reject and retry if guardrails fail.

Fallback & Degradation

What if the LLM fails? Have fallbacks: (1) cached response, (2) smaller/faster model, (3) hardcoded response, (4) human escalation. Graceful degradation beats hard failure.

import json, time
from dataclasses import dataclass, field
from openai import OpenAI
from pathlib import Path

client = OpenAI()

@dataclass
class ReviewItem:
    id: str
    input: str
    output: str
    flag_reason: str
    timestamp: float = field(default_factory=time.time)
    reviewed: bool = False
    approved: bool = False

class ReviewQueue:
    def __init__(self, path: str = "review_queue.jsonl"):
        self.path = Path(path)

    def flag(self, item: ReviewItem):
        with self.path.open("a") as f:
            f.write(json.dumps(item.__dict__) + "
")
        print(f"[QUEUE] Flagged: {item.flag_reason} — {item.input[:50]}")

    def pending(self) -> list[dict]:
        if not self.path.exists(): return []
        return [json.loads(l) for l in self.path.read_text().splitlines()
                if not json.loads(l).get("reviewed")]

queue = ReviewQueue()

def safe_generate(user_input: str, system: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "system", "content": system},
                  {"role": "user", "content": user_input}]
    ).choices[0].message.content

    # Flag if output looks sensitive
    if any(w in resp.lower() for w in ["sorry", "cannot", "unsafe", "harmful"]):
        queue.flag(ReviewItem(
            id=f"r-{int(time.time())}",
            input=user_input, output=resp,
            flag_reason="potential_refusal_or_safety"
        ))
    return resp