/ai-infrastructure-litellm

name: ai-infrastructure-litellm
description: LiteLLM proxy server setup, TypeScript client patterns via OpenAI SDK, model routing, fallbacks, load balancing, spend tracking, virtual keys, and production deployment

LiteLLM Proxy Patterns

> **Quick Guide:** LiteLLM is an OpenAI-compatible proxy (AI gateway) that routes requests to 100+ LLM providers. TypeScript clients connect via the standard OpenAI SDK with `baseURL` pointed at the proxy. Configure models, fallbacks, load balancing, and budgets in `config.yaml`. Use `provider/model-name` format in `litellm_params.model` (e.g., `anthropic/claude-sonnet-4-20250514`). The `model_name` in config is the user-facing alias clients request. Virtual keys require PostgreSQL. Master key must start with `sk-`.

---

<critical_requirements>

CRITICAL: Before Using This Skill

**(You MUST use the `provider/model-name` format in `litellm_params.model` -- e.g., `anthropic/claude-sonnet-4-20250514`, `openai/gpt-4o`, `azure/my-deployment` -- the provider prefix is how LiteLLM routes to the correct API)**

**(You MUST set `model_name` as the user-facing alias that clients request -- this is NOT the provider model ID, it is the name your TypeScript client passes as `model`)**

**(You MUST point the OpenAI SDK `baseURL` at the proxy URL (e.g., `http://localhost:4000`) and pass the proxy key as `apiKey` -- do NOT use provider API keys directly in client code)**

**(You MUST start master keys with `sk-` -- LiteLLM rejects master keys that do not follow this prefix convention)**

**(You MUST configure `database_url` pointing to PostgreSQL before using virtual keys, spend tracking, or team/user management -- these features require persistent storage)**

</critical_requirements>

---

**Auto-detection:** LiteLLM, litellm, litellm_params, litellm_settings, LLM proxy, LLM gateway, model_list, master_key, virtual keys, model fallback, load balancing LLM, provider/model, anthropic/claude, openai/gpt, azure/, litellm --config, LITELLM_MASTER_KEY, LITELLM_SALT_KEY

**When to use:**

• Running a unified LLM gateway that routes to multiple providers (OpenAI, Anthropic, Azure, Bedrock, etc.)
• Configuring model fallbacks, load balancing, or routing strategies across deployments
• Managing API key access with virtual keys, per-key budgets, and rate limits
• Tracking spend across models, teams, users, and tags
• Deploying a self-hosted OpenAI-compatible proxy with Docker

**Key patterns covered:**

• Proxy server config.yaml structure (model_list, litellm_settings, router_settings, general_settings)
• TypeScript client setup via OpenAI SDK pointed at proxy
• Model routing with provider prefixes and user-facing aliases
• Fallback chains (regular, context window, content policy, default)
• Load balancing strategies (simple-shuffle, least-busy, usage-based, latency-based, cost-based)
• Virtual keys with budgets, rate limits, and model restrictions
• Spend tracking per key, user, team, and tag
• Docker Compose production deployment

**When NOT to use:**

• Calling a single LLM provider directly with no proxy layer -- use the provider's SDK directly
• Building a Python application that calls LiteLLM as a library -- this skill covers the proxy server + TypeScript client pattern
• When you need framework-specific chat UI hooks -- use a framework-integrated AI SDK

---

Examples Index

• [Core: Config & Client Setup](examples/core.md) -- config.yaml structure, TypeScript OpenAI SDK client, model routing, Docker deployment
• [Routing & Reliability](examples/routing.md) -- Fallbacks, load balancing, cooldowns, retries, priority routing
• [Keys & Spend](examples/keys-and-spend.md) -- Virtual keys, budgets, rate limits, spend tracking, team management

---

<philosophy>

Philosophy

LiteLLM Proxy is an **AI gateway** -- a single OpenAI-compatible endpoint that routes to 100+ LLM providers. TypeScript applications never talk to providers directly; they talk to the proxy using the standard OpenAI SDK.

**Core principles:**

1. **Provider abstraction** -- Client code uses a single `baseURL` and standard OpenAI SDK. Switching providers means changing `config.yaml`, not application code. 2. **Two-layer naming** -- `model_name` is what clients request (e.g., `"claude-sonnet"`). `litellm_params.model` is the actual provider routing (e.g., `"anthropic/claude-sonnet-4-20250514"`). This decouples client code from provider specifics. 3. **Resilience via config** -- Fallbacks, retries, load balancing, and cooldowns are all declared in `config.yaml`. No application-level retry logic needed. 4. **Spend governance** -- Virtual keys, per-key budgets, rate limits, and tag-based tracking give fine-grained cost control without changing client code. 5. **OpenAI compatibility** -- Any client, SDK, or tool that works with OpenAI's API works with LiteLLM. No custom SDK required.

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Minimal config.yaml

The proxy needs a `config.yaml` with at least one model defined. `model_name` is client-facing; `litellm_params.model` is the provider route.

# config.yaml
model_list:
  - model_name: claude-sonnet # What clients request
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514 # Provider/model route
      api_key: os.environ/ANTHROPIC_API_KEY # Never hardcode keys

  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

**Why good:** Two-layer naming decouples clients from providers, `os.environ/` syntax reads secrets from environment at runtime

# BAD: Missing provider prefix, hardcoded key
model_list:
  - model_name: claude-sonnet-4-20250514 # Using provider model ID as name
    litellm_params:
      model: claude-sonnet-4-20250514 # No provider prefix -- routing fails
      api_key: sk-ant-abc123 # Hardcoded API key

**Why bad:** Without `anthropic/` prefix, LiteLLM cannot route to the correct provider; hardcoded keys are a security r