ai-infrastructure-litellm

LiteLLM Proxy Patterns

Quick Guide: LiteLLM is an OpenAI-compatible proxy (AI gateway) that routes requests to 100+ LLM providers. TypeScript clients connect via the standard OpenAI SDK with baseURL pointed at the proxy. Configure models, fallbacks, load balancing, and budgets in config.yaml. Use provider/model-name format in litellm_params.model (e.g., anthropic/claude-sonnet-4-20250514). The model_name in config is the user-facing alias clients request. Virtual keys require PostgreSQL. Master key must start with sk-.

<critical_requirements>

CRITICAL: Before Using This Skill

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST use the provider/model-name format in litellm_params.model -- e.g., anthropic/claude-sonnet-4-20250514, openai/gpt-4o, azure/my-deployment -- the provider prefix is how LiteLLM routes to the correct API)

(You MUST set model_name as the user-facing alias that clients request -- this is NOT the provider model ID, it is the name your TypeScript client passes as model)

(You MUST point the OpenAI SDK baseURL at the proxy URL (e.g., http://localhost:4000) and pass the proxy key as apiKey -- do NOT use provider API keys directly in client code)

(You MUST start master keys with sk- -- LiteLLM rejects master keys that do not follow this prefix convention)

(You MUST configure database_url pointing to PostgreSQL before using virtual keys, spend tracking, or team/user management -- these features require persistent storage)

</critical_requirements>

Auto-detection: LiteLLM, litellm, litellm_params, litellm_settings, LLM proxy, LLM gateway, model_list, master_key, virtual keys, model fallback, load balancing LLM, provider/model, anthropic/claude, openai/gpt, azure/, litellm --config, LITELLM_MASTER_KEY, LITELLM_SALT_KEY

When to use:

• Running a unified LLM gateway that routes to multiple providers (OpenAI, Anthropic, Azure, Bedrock, etc.)
• Configuring model fallbacks, load balancing, or routing strategies across deployments
• Managing API key access with virtual keys, per-key budgets, and rate limits
• Tracking spend across models, teams, users, and tags
• Deploying a self-hosted OpenAI-compatible proxy with Docker

Key patterns covered:

• Proxy server config.yaml structure (model_list, litellm_settings, router_settings, general_settings)
• TypeScript client setup via OpenAI SDK pointed at proxy
• Model routing with provider prefixes and user-facing aliases
• Fallback chains (regular, context window, content policy, default)
• Load balancing strategies (simple-shuffle, least-busy, usage-based, latency-based, cost-based)
• Virtual keys with budgets, rate limits, and model restrictions
• Spend tracking per key, user, team, and tag
• Docker Compose production deployment

When NOT to use:

• Calling a single LLM provider directly with no proxy layer -- use the provider's SDK directly
• Building a Python application that calls LiteLLM as a library -- this skill covers the proxy server + TypeScript client pattern
• When you need framework-specific chat UI hooks -- use a framework-integrated AI SDK

Examples Index

• [Core: Config & Client Setup](examples/core.md) -- config.yaml structure, TypeScript OpenAI SDK client, model routing, Docker deployment
• [Routing & Reliability](examples/routing.md) -- Fallbacks, load balancing, cooldowns, retries, priority routing
• [Keys & Spend](examples/keys-and-spend.md) -- Virtual keys, budgets, rate limits, spend tracking, team management

<philosophy>

Philosophy

LiteLLM Proxy is an AI gateway -- a single OpenAI-compatible endpoint that routes to 100+ LLM providers. TypeScript applications never talk to providers directly; they talk to the proxy using the standard OpenAI SDK.

Core principles:

1. Provider abstraction -- Client code uses a single baseURL and standard OpenAI SDK. Switching providers means changing config.yaml, not application code.
2. Two-layer naming -- model_name is what clients request (e.g., "claude-sonnet"). litellm_params.model is the actual provider routing (e.g., "anthropic/claude-sonnet-4-20250514"). This decouples client code from provider specifics.
3. Resilience via config -- Fallbacks, retries, load balancing, and cooldowns are all declared in config.yaml. No application-level retry logic needed.
4. Spend governance -- Virtual keys, per-key budgets, rate limits, and tag-based tracking give fine-grained cost control without changing client code.
5. OpenAI compatibility -- Any client, SDK, or tool that works with OpenAI's API works with LiteLLM. No custom SDK required.

</philosophy>

<patterns>

Core Patterns

Pattern 1: Minimal config.yaml

The proxy needs a config.yaml with at least one model defined. model_name is client-facing; litellm_params.model is the provider route.

# config.yaml
model_list:
  - model_name: claude-sonnet # What clients request
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514 # Provider/model route
      api_key: os.environ/ANTHROPIC_API_KEY # Never hardcode keys

  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

Why good: Two-layer naming decouples clients from providers, os.environ/ syntax reads secrets from environment at runtime

# BAD: Missing provider prefix, hardcoded key
model_list:
  - model_name: claude-sonnet-4-20250514 # Using provider model ID as name
    litellm_params:
      model: claude-sonnet-4-20250514 # No provider prefix -- routing fails
      api_key: sk-ant-abc123 # Hardcoded API key

Why bad: Without anthropic/ prefix, LiteLLM cannot route to the correct provider; hardcoded keys are a security risk; using the provider model ID as model_name couples clients to provider naming

See: [examples/core.md](examples/core.md) for complete config with general_settings, Docker setup

Pattern 2: TypeScript Client via OpenAI SDK

Connect to the proxy using the standard OpenAI SDK. Point baseURL at the proxy, use the proxy key as apiKey.

// lib/llm-client.ts
import OpenAI from "openai";

const PROXY_URL = "http://localhost:4000";

const client = new OpenAI({
  baseURL: PROXY_URL,
  apiKey: process.env.LITELLM_API_KEY, // Virtual key or master key
});

export { client };

// usage.ts
import { client } from "./lib/llm-client.js";

const completion = await client.chat.completions.create({
  model: "claude-sonnet", // model_name from config.yaml, NOT provider model ID
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Explain TypeScript generics." },
  ],
});

console.log(completion.choices[0].message.content);

Why good: Standard OpenAI SDK, no custom dependencies; model name matches config.yaml model_name; proxy key keeps provider keys server-side

// BAD: Using provider model ID, provider API key
const client = new OpenAI({
  baseURL: "http://localhost:4000",
  apiKey: process.env.ANTHROPIC_API_KEY, // Wrong -- use proxy key
});

const completion = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-4-20250514", // Wrong -- use model_name alias
  messages: [{ role: "user", content: "Hello" }],
});

Why bad: Provider API key bypasses proxy auth and virtual key controls; using provider model ID instead of alias couples client to provider naming and bypasses proxy routing logic

See: [examples/core.md](examples/core.md) for streaming, metadata tagging

Pattern 3: Fallback Chains

Configure model fallbacks so requests automatically retry on a different model when the primary fails.

# config.yaml
model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthrop