SGLang

What makes SGLang different

SGLang (Structured Generation Language) addresses two bottlenecks that standard LLM serving leaves on the table: redundant KV-cache computation and inefficient structured output generation.

RadixAttention: SGLang stores the KV-cache in a radix tree indexed by token sequence. When multiple requests share a common prefix (e.g. the same system prompt), the prefix's KV values are computed once and reused across all requests. This gives 3–5× throughput improvement on workloads with shared prefixes — common in agentic pipelines and batch classification tasks.

Compressed finite-state machine: For structured outputs (JSON, regex), SGLang pre-computes which tokens are valid at each generation step and masks the logits. This is faster than regex matching post-hoc and guarantees structural validity.

Deploying the SGLang server

pip install "sglang[all]"

# Start server with Llama 3 8B
python -m sglang.launch_server   --model-path meta-llama/Meta-Llama-3-8B-Instruct   --port 30000   --mem-fraction-static 0.85

# Multi-GPU with tensor parallelism
python -m sglang.launch_server   --model-path meta-llama/Meta-Llama-3-70B-Instruct   --tp 4   --port 30000

The server exposes an OpenAI-compatible API at http://localhost:30000/v1, so existing clients work unchanged.

import openai
client = openai.OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "What is 2+2?"}],
)
print(resp.choices[0].message.content)

RadixAttention and prefix caching

Every request's token sequence is stored as a path in a radix tree. When a new request arrives, SGLang finds the longest common prefix with cached sequences and reuses those KV values. Only the novel suffix needs computation.

# Example: all these requests share a 500-token system prompt
# Without prefix caching: each pays ~500 tokens of prefill
# With RadixAttention: first request pays 500 tokens, rest pay ~0 for prefix

requests = [
    [system_prompt, "Classify this email as spam/not-spam: " + email_1],
    [system_prompt, "Classify this email as spam/not-spam: " + email_2],
    [system_prompt, "Classify this email as spam/not-spam: " + email_3],
]
# SGLang detects the shared prefix and caches it after the first request

RadixAttention is most effective for: batch inference with shared prompts, multi-turn conversations (prefix grows with history), RAG pipelines where the retrieved context is reused, and agentic loops with fixed tool descriptions.

Structured generation

SGLang guarantees structurally valid outputs via constrained decoding. At each generation step, the server computes the set of valid next tokens given the current output and the constraint, then masks invalid tokens to zero probability before sampling.

import sglang as sgl

@sgl.function
def extract_entity(s, text):
    s += sgl.system("You are an entity extractor. Return JSON only.")
    s += sgl.user(f"Extract person, org, location from: {text}")
    # Constrain output to valid JSON matching this schema
    s += sgl.assistant(
        sgl.gen("result", max_tokens=200,
                json_schema={
                    "type": "object",
                    "properties": {
                        "person": {"type": "array", "items": {"type": "string"}},
                        "org":    {"type": "array", "items": {"type": "string"}},
                        "location": {"type": "array", "items": {"type": "string"}},
                    }
                })
    )
    return s["result"]

result = extract_entity.run(text="Tim Cook announced Apple's new product in Cupertino.")
print(result)  # Always valid JSON

The SGLang Python DSL

SGLang includes a Python DSL for writing multi-call LLM programs. Programs are compiled into optimised execution graphs that automatically parallelise independent calls and share KV-cache across dependent calls.

import sglang as sgl

@sgl.function
def multi_turn_qa(s, question):
    s += sgl.system("You are a helpful assistant.")
    s += sgl.user(question)
    s += sgl.assistant(sgl.gen("answer1", max_tokens=100))
    s += sgl.user("Now give a shorter version.")
    s += sgl.assistant(sgl.gen("answer2", max_tokens=50))

# Batch execution — runs requests concurrently, sharing prefix KV-cache
questions = ["Explain transformers", "What is RAG?", "What is LoRA?"]
states = sgl.function.run_batch(
    [{"question": q} for q in questions],
    progress_bar=True,
)
for q, s in zip(questions, states):
    print(f"Q: {q}")
    print(f"Full: {s['answer1']}")
    print(f"Short: {s['answer2']}")

Benchmarks

SGLang typically outperforms vLLM and TGI on workloads with shared prefixes. On pure throughput benchmarks without shared prefixes, performance is roughly comparable to vLLM (within 10–20%).

Key scenarios where SGLang excels: batch document processing (all requests share a long instruction prefix), RAG (documents are reused across queries), multi-turn agents (conversation history prefix grows and is reused), and structured generation (JSON/regex constraints are applied efficiently).

SGLang is maintained by the LMSYS team (same group behind Chatbot Arena) and is production-used at scale. It also supports speculative decoding with an eagle-style draft model for additional speedups.

Gotchas

Memory configuration: --mem-fraction-static controls what fraction of GPU memory is pre-allocated for KV-cache. If too high, the process OOMs loading the model. Default 0.9 is aggressive; start at 0.85.

Chunked prefill: For very long inputs, SGLang chunks prefill into smaller pieces to avoid GPU memory spikes. Enable with --chunked-prefill-size 4096.

Prefix caching requires deterministic tokenisation: If you format prompts differently between requests (extra spaces, different BOS handling), the cache won't hit. Use a consistent prompt template function.

DSL vs REST API: The Python DSL is powerful but adds a dependency on the SGLang Python client. For simple deployments, the OpenAI-compatible REST API is lower friction.