LLM Infrastructure

The Serving Stack

Serving LLMs in production is a distinct engineering problem from training them. The bottleneck is not compute — it is GPU memory bandwidth, request concurrency, and cost per token at scale.

The serving stack layers: (1) Inference engine (vLLM, TGI, SGLang) handles batching and KV cache. (2) Routing layer (LiteLLM, LLM Proxy) selects the right model. (3) Caching layer (Redis) caches tokens and embeddings. (4) Hardware (H100s, A100s, multi-GPU). (5) Cloud orchestration (Kubernetes, Ray Serve).

Bottleneck Shifts

Training: Compute-bound (FLOP/s). Inference (batch=1): Memory-bandwidth bound. Each token needs ~2 bytes × model params. Inference (batch=large): Compute-bound again (matrix multiplies overlap bandwidth stalls).

Inference Engines (vLLM, TGI, SGLang)

Modern inference engines solve the batching problem via continuous batching: as requests arrive, interleave token generation. This dramatically improves throughput compared to static batching (wait for batch to fill).

Popular Engines

# Serve a quantized Mistral-7B with continuous batching
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Mistral-7B-Instruct-v0.2-AWQ \
  --quantization awq \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096 \
  --served-model-name mistral-7b \
  --port 8000

# Quick health check
curl http://localhost:8000/health

# Test inference (OpenAI-compatible API)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "mistral-7b",
       "messages": [{"role": "user", "content": "Hello!"}],
       "max_tokens": 64, "temperature": 0.0}'

# Monitor GPU utilisation
watch -n1 nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total \
  --format=csv,noheader

Quantization (GPTQ, AWQ, GGUF)

Quantization reduces model weight precision (FP32 → INT8, INT4, or INT2). It cuts VRAM by 2–8× with ~5% accuracy loss. For large models (70B+), quantization is mandatory for single-GPU inference.

Quantization Methods

Quantization Frameworks

Hardware (GPUs, Memory Bandwidth)

GPU choice is driven by memory bandwidth, not peak FLOP/s. Inference is bandwidth-bound. An H100 is 5× faster than A100 for inference primarily because of 3× higher bandwidth (3.35 TB/s vs 1.9 TB/s).

GPU Comparison

Multi-GPU Inference

Data parallelism: Shard batch across GPUs. Simple but doesn't help single large request. Tensor parallelism: Shard model weights across GPUs. Each GPU computes part of each layer. Synchronization overhead ~5–10%. Pipeline parallelism: Split layers across GPUs. Good for sequential (one batch) but harder to saturate.

Cloud Deployment Patterns

Deploying LLMs to cloud requires: (1) Container (Docker) + orchestration (Kubernetes). (2) Auto-scaling based on queue depth. (3) Model serving framework (Seldon, KServe). (4) Monitoring + cost tracking.

import time, statistics
from vllm import LLM, SamplingParams

llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.2",
          tensor_parallel_size=1,
          max_model_len=4096)

prompts = ["Explain transformer attention in one paragraph."] * 32
params = SamplingParams(temperature=0.0, max_tokens=256)

# Warm-up run (JIT compilation)
_ = llm.generate(prompts[:2], params)

# Timed benchmark
start = time.perf_counter()
outputs = llm.generate(prompts, params)
elapsed = time.perf_counter() - start

total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
latencies = [elapsed / len(prompts) * 1000] * len(prompts)  # approx

print(f"Batch size   : {len(prompts)}")
print(f"Total time   : {elapsed:.2f}s")
print(f"Throughput   : {total_tokens/elapsed:.0f} tok/s")
print(f"Avg latency  : {elapsed/len(prompts)*1000:.0f} ms/req")
print(f"Cost est.    : ${total_tokens/1e6 * 0.20:.4f} (at $0.20/1M tok)")