Sync vs Async Serving

Sync vs async fundamentals

When a user sends an LLM request, your server must: receive the request, run inference (the slow part), and return the response. Synchronous serving means the worker thread blocks during inference — simple to reason about but wastes CPU cycles waiting for GPU work to finish. Asynchronous serving uses Python's asyncio or a similar event loop so the same thread can handle other requests while waiting for inference I/O to complete.

For LLM inference specifically, the GPU does most of the work and inference is compute-bound rather than I/O-bound. The main async win comes from batching multiple concurrent requests together, letting the inference engine amortise the fixed overhead of a forward pass over many tokens at once.

FastAPI async inference endpoint

from fastapi import FastAPI
from pydantic import BaseModel
import asyncio, time

app = FastAPI()

class InferRequest(BaseModel):
    prompt: str
    max_tokens: int = 256

# Async endpoint — doesn't block the event loop
@app.post("/generate")
async def generate(req: InferRequest):
    # run_in_executor offloads blocking inference to a thread pool
    # so the event loop stays free for health checks, other requests
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(
        None,           # default ThreadPoolExecutor
        _blocking_generate,
        req.prompt,
        req.max_tokens,
    )
    return {"text": result, "tokens": len(result.split())}

def _blocking_generate(prompt: str, max_tokens: int) -> str:
    # your actual model call here (transformers, vllm, etc.)
    time.sleep(0.5)  # simulate inference
    return f"Response to: {prompt[:50]}"

Streaming responses with SSE

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()

async def token_stream(prompt: str):
    # Yields tokens as they are produced — reduces time-to-first-token UX
    words = f"Streaming response to: {prompt}".split()
    for word in words:
        yield f"data: {word}\n\n"
        await asyncio.sleep(0.05)  # simulate per-token delay
    yield "data: [DONE]\n\n"

@app.get("/stream")
async def stream_generate(prompt: str = "Hello"):
    return StreamingResponse(
        token_stream(prompt),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

# Client-side consumption with httpx:
# async with client.stream("GET", "/stream", params={"prompt": "Hi"}) as r:
#     async for line in r.aiter_lines():
#         if line.startswith("data: ") and line != "data: [DONE]":
#             print(line[6:], end=" ", flush=True)

Thread pools for CPU-bound inference

Python's asyncio is designed for I/O-bound work. Model inference (even with GPU) involves Python overhead that holds the GIL. Use these patterns to prevent blocking the event loop:

• ThreadPoolExecutor: Default approach for inference. FastAPI's run_in_executor uses this. Keep pool size ≤ number of GPU replicas.
• ProcessPoolExecutor: For true CPU-bound preprocessing. Bypasses the GIL but has higher IPC overhead.
• vLLM AsyncLLMEngine: The right tool for production LLM serving — built-in async, continuous batching, and streaming without any of this manual plumbing.

from concurrent.futures import ThreadPoolExecutor
import asyncio

executor = ThreadPoolExecutor(max_workers=4)  # 4 model replicas

async def async_infer(prompt: str) -> str:
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(executor, sync_model_call, prompt)

Throughput vs latency tradeoffs

The fundamental tension in LLM serving:

• Maximise throughput: Batch many requests together (continuous batching). Each request waits a little longer to start but the system handles more tokens/second overall. Best for offline batch jobs, embeddings APIs.
• Minimise latency: Process each request immediately. Low p50 but wasted GPU capacity when traffic is low. Best for interactive chat where p99 matters.
• Streaming: Orthogonal to the above — you can stream a batched response. Streaming improves perceived latency (user sees tokens appearing) without changing wall-clock completion time.

When to use each pattern

Use synchronous endpoints when: latency is more important than throughput (single-user apps, debugging), you're using a simple framework like Flask, or the inference is fast (<100ms) and parallelism isn't needed.

Use async endpoints when: you need to handle many concurrent requests, you're calling remote inference APIs (network I/O bound), or you want to combine inference with other async operations (database lookups, external APIs).

Use streaming when: the response is long and the user is watching — streaming makes a 10-second response feel responsive immediately. Most chat interfaces expect SSE or WebSocket streaming.

Gotchas

• Don't await blocking code: await slow_sync_function() does NOT make it async — it still blocks the event loop. Always use run_in_executor.
• Connection pooling: If you're calling a remote inference API, use a single async HTTP client (httpx.AsyncClient) with connection pooling rather than creating a new client per request.
• Backpressure: Without a queue, async frameworks silently accept thousands of concurrent requests. Add a semaphore or queue limit to avoid OOM.
• SSE buffering: Nginx and some reverse proxies buffer responses by default. Set X-Accel-Buffering: no to ensure streaming reaches the client immediately.