Timeout Budget

import time
from dataclasses import dataclass
@dataclass
class RequestContext:
    request_id: str
    deadline: float  # Unix timestamp
@property
    def remaining_ms(self) -> float:
        return max(0, (self.deadline - time.time()) * 1000)
def child_ctx(self, fraction: float) -> 'RequestContext':
        # Allocate a fraction of remaining budget to a sub-task
        remaining = self.deadline - time.time()
        return RequestContext(
            request_id=self.request_id,
            deadline=time.time() + remaining * fraction,
        )
async def pipeline(query: str, ctx: RequestContext) -> str:
    docs = await retrieve(query, ctx.child_ctx(0.1))   # 10% budget
    reranked = await rerank(docs, ctx.child_ctx(0.08)) # 8% budget
    return await generate(query, reranked, ctx)        # remainder

Timeout Implementation Patterns

Practical timeout implementation requires careful consideration of layered deadlines. System-level timeouts prevent resource exhaustion, middleware timeouts manage service latency SLAs, and application timeouts enable graceful degradation. Empirical studies show well-tuned timeout hierarchies reduce p99 latency by 40-60% while improving error rates.

# Hierarchical timeout implementation
from contextlib import asynccontextmanager
import asyncio
from functools import wraps

class TimeoutBudget:
    def __init__(self, total_ms):
        self.total_ms = total_ms
        self.start_time = None
        self.allocations = {}
    
    @asynccontextmanager
    async def allocate(self, component_name, allocation_percent):
        """Allocate timeout budget to component"""
        if self.start_time is None:
            self.start_time = asyncio.get_event_loop().time()
        
        allocation_ms = self.total_ms * (allocation_percent / 100)
        elapsed = (asyncio.get_event_loop().time() - self.start_time) * 1000
        remaining = allocation_ms - elapsed
        
        self.allocations[component_name] = remaining
        
        try:
            async with asyncio.timeout(remaining / 1000):
                yield
        except asyncio.TimeoutError:
            raise TimeoutError(f"{component_name} exceeded budget: {remaining}ms")

# Usage in LLM pipeline
async def llm_pipeline(user_query):
    budget = TimeoutBudget(total_ms=5000)  # 5 second total
    
    async with budget.allocate("retrieval", 30):  # 30% = 1500ms
        docs = await retrieve_documents(user_query)
    
    async with budget.allocate("generation", 60):  # 60% = 3000ms
        response = await generate_response(user_query, docs)
    
    async with budget.allocate("ranking", 10):  # 10% = 500ms
        ranked = await rank_candidates(response)
    
    return ranked

# Timeout observability and alerting
class TimeoutObserver:
    def __init__(self):
        self.metrics = {}
    
    def record_timeout(self, component, budget_ms, actual_ms, success):
        key = f"{component}_utilization"
        utilization = actual_ms / budget_ms if budget_ms > 0 else 0
        
        if key not in self.metrics:
            self.metrics[key] = []
        
        self.metrics[key].append(utilization)
        
        # Alert if consistently near limit
        recent = self.metrics[key][-100:]
        avg_util = sum(recent) / len(recent)
        
        if avg_util > 0.8:
            self.send_alert(f"{component} using {avg_util:.0%} of timeout budget")
    
    def get_utilization_report(self):
        """Generate dashboard data"""
        report = {}
        for key, values in self.metrics.items():
            report[key] = {
                'mean': sum(values) / len(values),
                'p95': sorted(values)[int(len(values) * 0.95)],
                'max': max(values)
            }
        return report

Graceful Degradation Under Load

When timeouts trigger, systems must gracefully degrade functionality. Common strategies include returning cached responses, using faster approximate models, reducing output quality (shorter summaries), or queuing requests. Empirical results show well-designed fallbacks maintain user satisfaction to 80%+ even with 10x traffic spikes.

Implementing timeout budgets requires principled allocation strategies across system components. The basic principle: total timeout should be aggressive enough (1-2x median latency) to provide SLA guarantees, while individual component budgets ensure one slow component doesn't block others. Allocation percentages depend on component criticality and variability: retrieval systems with under 100ms p50 latency might receive 30% of budget if they spike to 1s occasionally, while LLM generation with high p95 variance might receive 60% despite similar median latency. Streaming responses enable partial-result fallbacks: return top-3 results if ranking times out, stream response incrementally if generation exceeds soft timeout. Caching strategies reduce timeout necessity: cache retrieval results for 24 hours, cache common query responses indefinitely. Load shedding during traffic spikes prevents cascading timeouts: queue excess requests when system utilization exceeds 80%, serve queued requests from cache if primary servers unavailable. Observability is critical: instrument each component with latency histograms, measure timeout frequency, alert when components consistently approach budget limits. Production analysis shows well-tuned timeouts reduce 99th percentile latency by 30-40% while improving user experience through faster failure feedback (timeout messaging) instead of hanging requests.

Detailed timeout budget allocation depends on component latency distributions and criticality. P50 vs P95 latencies differ significantly: retrieval might have p50=50ms but p95=300ms due to cache misses, requiring larger budget allocation for tail latency. Percentile-based allocation: allocate based on p95 latency + buffer, not average latency. For 5-second total budget with components: retrieval (p95=400ms, allocation=600ms), LLM generation (p95=2s, allocation=2500ms), ranking (p95=200ms, allocation=400ms), overhead (500ms), leaves 100ms buffer for jitter. Soft timeouts (warn but continue) paired with hard timeouts (fail) improve results: LLM generation soft timeout at 2s returns incomplete response (still useful), hard timeout at 2.5s completely fails. Adaptive timeouts adjust based on load: under normal load use conservative timeouts, under high load extend slightly to avoid cascading failures. Hierarchical timeouts: system-level timeout 5s, service-level 4.5s, component-level budgets, ensuring failures are caught at lowest level. Timeout granularity: coarse timeouts (1 second buckets) tolerate 100ms variance, fine timeouts (10ms buckets) catch subtle regressions. Observability is critical: log all timeout events with components involved, causes, and fallback used. Analyze timeout patterns: high timeout rate in retrieval suggests database slowdown (add caching), high timeout in generation suggests model overload (scale up). Cost-benefit analysis: timeout overhead (returning partial results, fallback costs) vs benefit (predictable latency, better UX). Optimization: good timeouts eliminate long-tail latency (reduce p99 by 60%), improving user satisfaction despite slightly lower perfect-response rate.

Fallback mechanisms implement graceful degradation when components timeout. Cache-first strategy: if real-time generation times out, return best cached result (stale but fast). Approximate models: timeout LLM generation, use faster distilled model (3-5x faster, 1-2% quality drop). Streaming responses: timeout full response generation, return partial response (first few paragraphs), continue in background. Reduced scope: timeout full retrieval, return limited subset (top-1 instead of top-5). Quality reduction: timeout high-quality synthesis, use lower-quality but faster variant. These fallbacks trade quality for predictability: average quality might drop 5-10% but p99 latency drops 40-60%, user experience often improves due to consistency. Fallback selection requires careful design: use fallback only when necessary (real timeout), not for all requests. Learning which fallbacks work best: analyze success rates (do users find partial responses useful?), user satisfaction (ratings, click-through), engagement metrics. Hybrid approaches: use cheap models initially, expensive models for slow requests, fallback to cheap if expensive times out. Implementation complexity: require instrumentation throughout stack, clear error handling, fallback specification per endpoint, graceful degradation logic. Testing fallbacks: include timeout scenarios in test suite, verify fallback paths execute correctly, measure latency of fallback codepaths. Monitoring: track fallback frequency per component, alert if fallbacks triggered >1% of requests.

Timeout budget design for multi-service systems requires accounting for cascading delays. Sequential components (A → B → C) accumulate timeouts: if A takes p95 200ms, B takes 300ms, C takes 200ms, total p95 is ~700ms (not sum due to percentile arithmetic—p95 of sum < sum of p95s). Parallel components (A, B, C in parallel) take p95 of max latency: if all 3 take 300ms p95, parallel takes 300ms (not 900ms). Resource contention causes timeout spikes: all components using same GPU memory, competing for cache, disk I/O saturation. Timeout budgets must account for contention: add 30-50% buffer for peak load scenarios. Soft-hard timeout combinations: soft timeout (1.8s) returns partial response, hard timeout (2.0s) fails fast. Enables SLA compliance (95% <1.8s) while catching runaway processes (hard timeout prevents 10+ second hangs). Connection pooling for network timeouts: reuse connections (setup cost 50-100ms), pool size matches expected concurrency. DNS resolution caching prevents 100-500ms delays on name lookups. Client-side retry strategies with exponential backoff: initial retry after 10ms, 100ms, 1s prevents retry storms on overloaded servers. Deadline propagation in distributed systems: RPC deadline passed from client → server → downstream services, servers check remaining time before accepting requests (reject if remaining < minimum required). Time synchronization (NTP) keeps clocks aligned across cluster, preventing timeout inaccuracies from skewed clocks. Observability: histogram of latencies per component (not just average), percentile-based SLAs (p50, p95, p99), alert on sustained threshold breaches.