Cloud Deployment for LLMs

Managed API vs Self-Hosted

Managed LLM APIs (OpenAI, Anthropic, Bedrock) trade control for simplicity. You send a request; they handle infrastructure, scaling, model updates. Self-hosted models (LLaMA, Mistral) run on your infrastructure. You control fine-tuning, caching, data residency, cost.

AWS Options

1. SageMaker Real-Time Endpoints

Managed endpoint serving. Deploy any model; SageMaker handles autoscaling and load balancing. Good for production workloads with consistent traffic. Pricing: you pay for instance time (e.g., ml.p4d.24xlarge at $37/hour). Cold starts are fast; scaling is automatic.

2. EKS + vLLM

Kubernetes on AWS. Deploy vLLM (high-throughput LLM inference engine) on EKS clusters. More control; scales via Karpenter. Better for burstable traffic and cost-sensitive workloads. Requires DevOps expertise.

3. Inferentia2 Chips

AWS custom silicon. Optimized for inference; ~70% cheaper than GPUs for throughput workloads. Tradeoff: limited model compatibility. Good for specific models (BERT, smaller LLMs).

4. AWS Bedrock

Managed API. Claude, Llama, Mixtral available via API. No infrastructure to manage. Per-token pricing. Best for teams wanting API convenience with data residency on AWS.

Python boto3 Example

GCP Options

1. Vertex AI Model Garden

GCP's hosted LLM platform. Deploy fine-tuned versions of Claude, Gemini, Llama. Autoscaling, monitoring included. Simpler than raw GKE; less control than containers.

2. GKE + vLLM

Google Kubernetes Engine. Deploy vLLM for maximum throughput and cost control. Scale with Workload Identity for IAM. Good for high-volume batch inference and real-time serving.

3. TPU v4/v5e

Google custom silicon. Extremely fast for LLMs optimized for TPUs. v5e is newer, cheaper; v4 is proven. Requires model optimization; not all models support TPUs well.

4. Cloud Run with Models

Serverless containers. Deploy lightweight models (ORCA, fine-tuned small models) to Cloud Run for autoscaling without managing servers. Cold starts ~5 sec.

Azure Options

1. Azure OpenAI Service

Managed API for GPT-4, GPT-3.5, embeddings. HIPAA/FedRAMP compliant. Per-token pricing. No infrastructure to manage. Best for enterprises needing Microsoft integration and compliance.

2. AKS + Triton

Azure Kubernetes Service with NVIDIA Triton Inference Server. Full control over model serving, batching, GPU optimization. Good for complex serving pipelines and custom kernels.

3. NC-series (H100) VMs

Virtual machines with H100 GPUs. Pay-as-you-go or reserved instances (30% discount). Deploy vLLM or TGI. More expensive than AWS/GCP but with Azure ecosystem integration.

4. Azure Machine Learning

Managed inference endpoints. Deploy Hugging Face models, fine-tuned models, ONNX. Autoscaling, monitoring included. Simpler than raw VMs; less flexible than AKS.

Auto-Scaling Patterns

Horizontal Pod Autoscaler (HPA)

Scale based on GPU utilization. If average GPU use > 70%, spawn new pods. Works well for steady traffic; slower for traffic spikes (30–60 sec to provision).

Queue-Depth Scaling

Scale based on request queue length, not resource use. If queue length > 10, scale out. Faster response to spikes but requires queue implementation (AWS SQS, Kubernetes custom metric).

Karpenter for Node Provisioning

Open-source auto-scaler for Kubernetes. Provisions optimal node types (GPU, CPU, spot). Faster than HPA; integrates with spot instances for 50–80% cost savings. Requires AWS or GCP.

YAML Example: HPA on GPU Utilization

import time, logging
from openai import OpenAI
import anthropic

logger = logging.getLogger(__name__)

class MultiCloudRouter:
    """Route LLM requests across providers with latency tracking and failover."""
    def __init__(self):
        self.providers = {
            "openai":    {"client": OpenAI(), "model": "gpt-4o-mini",
                          "latencies": [], "errors": 0},
            "anthropic": {"client": anthropic.Anthropic(), "model": "claude-haiku-4-5-20251001",
                          "latencies": [], "errors": 0},
        }

    def _call_openai(self, p: dict, prompt: str) -> str:
        return p["client"].chat.completions.create(
            model=p["model"],
            messages=[{"role": "user", "content": prompt}],
            max_tokens=256
        ).choices[0].message.content

    def _call_anthropic(self, p: dict, prompt: str) -> str:
        return p["client"].messages.create(
            model=p["model"],
            max_tokens=256,
            messages=[{"role": "user", "content": prompt}]
        ).content[0].text

    def route(self, prompt: str, prefer_fast: bool = True) -> dict:
        # Sort providers by recent avg latency
        def avg_lat(name):
            lats = self.providers[name]["latencies"][-5:]
            return sum(lats)/len(lats) if lats else float('inf')

        order = sorted(self.providers.keys(), key=avg_lat) if prefer_fast                 else list(self.providers.keys())

        for name in order:
            p = self.providers[name]
            t0 = time.perf_counter()
            try:
                call_fn = self._call_openai if name == "openai" else self._call_anthropic
                result = call_fn(p, prompt)
                latency = time.perf_counter() - t0
                p["latencies"].append(latency)
                return {"provider": name, "response": result, "latency_ms": round(latency*1000)}
            except Exception as e:
                p["errors"] += 1
                logger.warning(f"{name} failed: {e}, trying next provider")
        raise RuntimeError("All providers failed")

router = MultiCloudRouter()
result = router.route("What is 42 * 37?")
print(f"[{result['provider']}] {result['latency_ms']}ms: {result['response']}")

Cost Optimisation

Spot Instances & Preemptible VMs

AWS spot / GCP preemptible instances cost 50–80% less than on-demand. Tradeoff: can be terminated with 2-min notice. Works for stateless inference; use with Karpenter for seamless failover.

Reserved Instances

Commit to 1-year or 3-year terms; save 30–50% on instance costs. Lock in capacity; requires demand forecasting. Good for baseline traffic.

Right-Sizing

Use smallest GPU type that meets latency SLA. A100 is overkill for many models; try H100, L40S, or even A10G. Monitor GPU utilization; oversized instances waste money.

Batching Requests

Batch inference multiplies throughput. Instead of 10 sequential requests (10 sec), batch them (1 sec for 100 requests). Trade latency for throughput; good for async workloads.

Speculative Decoding & Caching

Speculative decoding reduces tokens generated (faster, cheaper). Prompt caching (Claude, Gemini) reuses computation on repeated context (30–50% token savings).

Tools & Platforms

References

• Docs AWS SageMaker — Real-time Endpoints. aws.amazon.com/sagemaker ↗
• Docs GCP Vertex AI — Model Serving. cloud.google.com/vertex-ai ↗
• Docs Azure ML — Inference Endpoints. azure.microsoft.com/ml ↗
• Docs vLLM — LLM Inference Library. docs.vllm.ai ↗
• Docs Karpenter — Auto-scaling for K8s. karpenter.sh ↗

• Blog Together.ai. (2024). Deploying LLMs Cost-Effectively. — together.ai ↗
• Blog Hugging Face. (2023). Inference at Scale: LLM Deployment Strategies. — huggingface.co/blog ↗