Circuit Breaker

What circuit breakers solve

When an upstream service (e.g. an LLM provider API) degrades or goes down, naive retry logic causes a cascade: threads pile up waiting for timeouts, memory fills with queued requests, and your entire service grinds to a halt even for requests that don't need the failing service. The circuit breaker pattern detects failure and fast-fails subsequent requests immediately, giving the upstream time to recover while keeping your service responsive.

Three states explained

• Closed (normal): Requests flow through. The circuit tracks the failure rate over a rolling window. When failures exceed the threshold, the circuit opens.
• Open (failing): All requests are immediately rejected with a fallback response — no actual calls made to the upstream. After a configured timeout, the circuit moves to half-open.
• Half-open (probing): A single probe request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens and the timeout resets.

Python implementation

import time, threading
from enum import Enum
from dataclasses import dataclass, field

class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5      # failures in window to open
    recovery_timeout: float = 30.0  # seconds before half-open probe
    window_size: int = 10           # rolling window for failure rate

    _state: State = field(default=State.CLOSED, init=False)
    _failures: int = field(default=0, init=False)
    _last_failure_time: float = field(default=0.0, init=False)
    _lock: threading.Lock = field(default_factory=threading.Lock, init=False)

    def call(self, func, *args, **kwargs):
        with self._lock:
            if self._state == State.OPEN:
                if time.time() - self._last_failure_time > self.recovery_timeout:
                    self._state = State.HALF_OPEN
                else:
                    raise RuntimeError("Circuit breaker OPEN — fast failing")

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise

    def _on_success(self):
        with self._lock:
            self._failures = 0
            self._state = State.CLOSED

    def _on_failure(self):
        with self._lock:
            self._failures += 1
            self._last_failure_time = time.time()
            if self._failures >= self.failure_threshold:
                self._state = State.OPEN

    @property
    def state(self) -> str:
        return self._state.value

Integrating with LLM API calls

import openai
from functools import wraps

cb = CircuitBreaker(failure_threshold=3, recovery_timeout=60.0)

def call_llm(prompt: str) -> str:
    def _inner():
        client = openai.OpenAI()
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            timeout=10.0,
        )
        return resp.choices[0].message.content

    try:
        return cb.call(_inner)
    except RuntimeError as e:
        if "OPEN" in str(e):
            # Fallback: cached response, simpler model, or error message
            return "Service temporarily unavailable. Please try again shortly."
        raise

# Usage
response = call_llm("Summarise this document...")
print(f"Circuit state: {cb.state}")

Thresholds and tuning

The right thresholds depend on your SLA and traffic pattern. As a starting point:

• failure_threshold: 3–5 failures is appropriate for LLM APIs. Too low causes false trips on transient errors; too high lets failures pile up.
• recovery_timeout: 30–60 seconds for LLM providers. Most transient rate-limit windows are 60 seconds; network blips resolve faster.
• Error discrimination: Don't count 4xx client errors (bad prompts, content policy) as circuit failures — only count 5xx server errors and timeouts.

Monitoring circuit state

import prometheus_client as prom

circuit_state_gauge = prom.Gauge(
    "llm_circuit_breaker_state",
    "Circuit breaker state (0=closed, 1=open, 2=half_open)",
    ["service"],
)

# Update metric on each call
state_map = {"closed": 0, "open": 1, "half_open": 2}
circuit_state_gauge.labels(service="openai").set(state_map[cb.state])

Alert when the circuit stays open for >5 minutes — that indicates a prolonged provider outage, not just a transient error.

Gotchas

• Thread safety: Circuit breaker state is shared across threads. Always use a lock around state transitions.
• Don't trip on timeouts you caused: If you set an aggressive 2s timeout and the model needs 3s, you'll trip the breaker on healthy infrastructure. Set timeouts generously.
• Fallback quality: The open-state fallback is the most visible part of circuit breaker design. A bad fallback ("error 503") is worse than a graceful degraded response.
• Libraries: Consider pybreaker or tenacity for production use rather than rolling your own.