OpenAI Compat API

What is the OpenAI-compatible API

OpenAI's Chat Completions API became the de facto standard for LLM inference after GPT-3.5 was released. The core endpoints (/v1/chat/completions, /v1/completions, /v1/embeddings) are now implemented by virtually every LLM serving framework: vLLM, Ollama, TGI, LM Studio, Llamafile, SGLang, and third-party providers (Groq, Together AI, Fireworks AI, Perplexity). If your code uses the OpenAI Python SDK, you can redirect it to any compatible server by changing the base_url — no other code changes needed.

Core endpoints

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",  # point to any compatible server
    api_key="not-needed-for-local",       # required by SDK, can be any string
)

# 1. Chat completions
resp = client.chat.completions.create(
    model="llama3",    # model name as registered in the server
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=256,
    stream=False,
)
print(resp.choices[0].message.content)

# 2. Streaming
for chunk in client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Count to 5"}],
    stream=True,
):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

# 3. Embeddings
emb = client.embeddings.create(
    model="nomic-embed-text",
    input=["Hello world", "Goodbye world"],
)
vectors = [e.embedding for e in emb.data]
print(f"Embedding dim: {len(vectors[0])}")

Switching providers with base_url

import os

# OpenAI
openai_client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Groq (fast inference, OpenAI-compatible)
groq_client = openai.OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

# Together AI
together_client = openai.OpenAI(
    base_url="https://api.together.xyz/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)

# Local Ollama
ollama_client = openai.OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

# Local vLLM
vllm_client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm",
)

# All clients use the same API — swap by changing the client object
def generate(client, model: str, prompt: str) -> str:
    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256,
    ).choices[0].message.content

LiteLLM for multi-provider routing

pip install litellm

import litellm

# LiteLLM normalises all providers to the OpenAI format
# Prefix model string with provider: "anthropic/", "groq/", etc.

response = litellm.completion(
    model="gpt-4o-mini",            # OpenAI
    messages=[{"role": "user", "content": "Hi"}],
)
response = litellm.completion(
    model="anthropic/claude-3-haiku-20240307",  # Anthropic
    messages=[{"role": "user", "content": "Hi"}],
)
response = litellm.completion(
    model="groq/llama3-8b-8192",   # Groq
    messages=[{"role": "user", "content": "Hi"}],
)

# LiteLLM proxy: OpenAI-compatible server that routes to any provider
# litellm --model gpt-4o-mini --port 8000
# Then point any OpenAI SDK client at http://localhost:8000/v1

Running locally with Ollama

# Install Ollama: https://ollama.com/download
ollama serve   # starts OpenAI-compatible server on localhost:11434
ollama pull llama3.2  # download a model

# Test with curl
curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}]}'

client = openai.OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is 2+2?"}],
)
print(resp.choices[0].message.content)

vLLM OpenAI-compatible server

# Start vLLM with OpenAI-compatible API
vllm serve meta-llama/Llama-3-8B-Instruct \
    --port 8000 \
    --api-key your-secret-key \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key",
)
# Model name = the HuggingFace model ID
resp = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Summarise transformers in 2 sentences."}],
)
print(resp.choices[0].message.content)

Gotchas

• Not 100% compatible: "OpenAI-compatible" means the core chat completions endpoint works, but provider-specific features (function calling, logprobs, vision, tool_choice) may not be fully supported by all servers. Test your specific use case against each endpoint.
• Model name is server-specific: The model string you pass is defined by the server, not by OpenAI. Ollama uses short names like llama3.2; vLLM uses the HuggingFace repo ID; Groq uses API-specific names. Check each server's docs.
• Rate limits and error formats: Error response formats differ across providers. OpenAI returns JSON error objects with specific codes; local servers may return different formats. Your error handling code may need to handle both.