Latency Budget

Two latency metrics: TTFT and TPOT

LLM latency has two distinct components that require different optimisations:

TTFT (Time to First Token): The time from when the request is sent until the first output token is received. This is dominated by: network round-trip + prefill computation (processing the input prompt). For a 1000-token prompt, prefill takes 0.5–2s on typical API infrastructure. This is what the user experiences as "the model thinking".

TPOT (Time Per Output Token): The time to generate each subsequent token after the first. Typically 20–80ms/token for frontier models, giving 12–50 tokens/second. For a 500-token response, this adds 10–40 seconds of total generation time.

For interactive applications, optimising TTFT improves perceived responsiveness more than TPOT — users tolerate slow token generation more than a long initial pause. Streaming output makes TPOT less important for user experience.

Decomposing the latency budget

A typical RAG application has these latency components. Set a budget for each based on your overall SLA:

If any component consistently exceeds its budget, focus optimisation there before elsewhere.

Measuring each component

import time
from functools import wraps

def latency_tracker(component_name: str):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            start = time.perf_counter()
            result = await func(*args, **kwargs)
            elapsed_ms = (time.perf_counter() - start) * 1000
            # Log to your metrics system
            metrics.histogram(f"latency.{component_name}", elapsed_ms,
                              tags={"p50": ..., "p99": ...})
            if elapsed_ms > BUDGET[component_name]:
                logger.warning(f"{component_name} exceeded budget: {elapsed_ms:.0f}ms")
            return result
        return wrapper
    return decorator

@latency_tracker("embedding")
async def embed_query(text: str) -> list:
    return await embed_client.create(text)

@latency_tracker("vector_search")
async def search(embedding: list, k: int = 5) -> list:
    return await vector_store.query(embedding, top_k=k)

@latency_tracker("llm_generation")
async def generate(prompt: str) -> str:
    chunks = []
    async for chunk in llm_client.stream(prompt):
        chunks.append(chunk)
        if len(chunks) == 1:
            metrics.record("ttft_ms", (time.perf_counter() - start) * 1000)
    return "".join(chunks)

Optimisation levers

Embedding latency: Cache query embeddings (same query = same vector). Use a local embedding model (no network round-trip) for queries — embedding models are small (100–400MB). Batch embedding requests at ingestion time.

Vector search: Approximate Nearest Neighbor (ANN) indices (HNSW, IVF) trade slight recall for 10–100× faster search vs exact search. Pre-filter by metadata before vector search to reduce the search space.

LLM TTFT: Reduce prompt length (shorter system prompts, fewer retrieved chunks). Use speculative decoding. Use a smaller model if quality allows. Cache common prompt prefixes (KV-cache warming in TGI/vLLM).

LLM generation: Use streaming — don't wait for full generation. Limit max_tokens aggressively. Use faster/smaller models for tasks where quality requirements are lower.

System-level: Use async throughout (no blocking I/O). Parallelize embedding + metadata lookup. Keep your application server co-located with the LLM API region.

Caching strategies

import hashlib
from functools import lru_cache
import redis

r = redis.Redis()

def cache_key(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:16]

async def embed_with_cache(text: str) -> list:
    key = f"emb:{cache_key(text)}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    embedding = await embed_client.create(text)
    r.setex(key, 3600, json.dumps(embedding))  # 1 hour TTL
    return embedding

# Semantic cache for LLM responses
async def query_with_semantic_cache(query: str, threshold: float = 0.95) -> str:
    query_emb = await embed_with_cache(query)
    # Search for similar past queries
    similar = await semantic_cache_store.search(query_emb, top_k=1)
    if similar and similar[0].similarity > threshold:
        return similar[0].cached_response  # reuse cached answer
    # Cache miss: run full RAG pipeline
    response = await full_rag_pipeline(query)
    await semantic_cache_store.store(query_emb, query, response)
    return response

P99 vs mean

Always track P99 (99th percentile) latency, not mean. The mean hides outliers that cause user-visible timeout errors. In LLM systems, P99 is often 3–5× the mean due to: occasional long-context requests hitting rate limits, cold model starts after scaling events, and occasional slow responses from LLM provider infrastructure.

Set SLAs in terms of P95 and P99: "95% of requests complete in <3s, 99% in <8s." Mean-based SLAs are meaningless for user-facing applications.

Tools: add latency histograms to your metrics (Prometheus histogram, Datadog distribution). Set alerting thresholds at P99, not mean. Log the full latency breakdown for any request exceeding the P99 budget to identify which component caused the outlier.

Gotchas

Async doesn't always help: Running embedding and LLM calls in parallel only helps if they're independent. In a standard RAG pipeline, you need the embedding before searching, and the search results before calling the LLM — these are sequential by nature. Parallelise what you can (e.g., retrieve while streaming previous tokens).

Context window affects TTFT nonlinearly: LLM prefill cost scales roughly quadratically with input length (due to attention). A 4K-token prompt takes ~4× longer to prefill than a 2K-token prompt. Reducing prompt length has outsized latency benefits for long-context applications.

Cold starts in serverless: If deploying on serverless (Lambda, Cloud Run), cold start times (200ms–2s) can dominate latency for infrequent users. Keep instances warm with periodic ping requests or use always-on minimum instances.