/ai-infrastructure-modal

name: ai-infrastructure-modal
description: Serverless GPU compute platform for AI model deployment — web endpoints, GPU functions, model serving, and TypeScript client patterns

Modal Patterns

> **Quick Guide:** Modal is a serverless GPU compute platform where you define Python functions with decorators and Modal handles containers, scaling, and GPU provisioning. TypeScript apps interact with Modal via HTTP endpoints (calling `@modal.fastapi_endpoint` or `@modal.asgi_app` functions) or the `modal` npm SDK (calling functions directly via gRPC). Define container images, secrets, and volumes as code -- no YAML config files. Use `modal deploy` for production, `modal serve` for dev.

---

<critical_requirements>

CRITICAL: Before Using This Skill

**(You MUST define Modal functions in Python -- the TypeScript SDK can call functions and manage resources but cannot define them)**

**(You MUST use `@modal.fastapi_endpoint` (not the old `@modal.web_endpoint`) for simple web endpoints -- renamed in Modal 1.0)**

**(You MUST use `modal.Volume` for model weight caching -- `@modal.build` is deprecated in Modal 1.0)**

**(You MUST never hardcode secrets in Modal code -- use `modal.Secret.from_name()` and access via `os.environ`)**

**(You MUST bind to `0.0.0.0` (not `127.0.0.1`) when using `@modal.web_server`)**

</critical_requirements>

---

**Auto-detection:** Modal, modal, modal.App, modal.Image, modal.Volume, modal.Secret, modal.gpu, modal.fastapi_endpoint, modal.asgi_app, modal.web_server, modal.Cron, modal.Period, modal deploy, modal serve, MODAL_TOKEN_ID, MODAL_TOKEN_SECRET, ModalClient

**When to use:**

• Deploying ML models (vLLM, Hugging Face, custom PyTorch) on serverless GPUs
• Creating HTTP API endpoints backed by GPU compute for TypeScript apps to consume
• Running scheduled GPU jobs (fine-tuning, batch inference, data processing)
• Calling Modal functions from TypeScript using the `modal` npm SDK
• Building AI inference pipelines with auto-scaling and pay-per-second billing

**Key patterns covered:**

• Web endpoints (`@modal.fastapi_endpoint`, `@modal.asgi_app`, `@modal.web_server`) for HTTP access
• TypeScript client patterns (fetch-based and `modal` npm SDK)
• Container images, secrets, volumes, and GPU configuration
• Model serving with vLLM and custom inference
• Scheduled functions and deployment

**When NOT to use:**

• Pure Python ML workloads with no TypeScript consumer -- this skill focuses on the TypeScript interaction surface
• Simple CPU-only tasks where a regular server or cloud function suffices
• When you need persistent long-running servers (Modal scales to zero by default)
• You need sub-100ms cold starts (Modal cold starts are 2-4 seconds)
• You need persistent WebSocket connections beyond request/response

---

Examples Index

• [Core: Web Endpoints & TypeScript Client](examples/core.md) -- Defining endpoints, calling from TypeScript, authentication, GPU functions, images, secrets, volumes
• [Quick API Reference](reference.md) -- CLI commands, decorator parameters, URL patterns, GPU types

---

<philosophy>

Philosophy

Modal eliminates infrastructure management for GPU workloads. Everything is code -- container images, GPU allocation, secrets, volumes, scaling rules. There are no YAML configs, Dockerfiles, or Kubernetes manifests.

**Core principles:**

1. **Infrastructure as Python code** -- Container images, GPU types, secrets, and volumes are all declared as Python decorators and objects. No separate config files. 2. **Serverless GPU scaling** -- Functions scale from zero to hundreds of GPUs automatically. You pay per second of compute, not for idle capacity. 3. **Two interaction models for TypeScript** -- Call Modal via HTTP endpoints (most common) or via the `modal` npm SDK for direct function invocation without HTTP overhead. 4. **Immutable deployments** -- `modal deploy` creates a named, persistent deployment with stable URLs. `modal serve` creates ephemeral dev endpoints.

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Web Endpoint (TypeScript Consumption)

The most common pattern: define a Python endpoint on Modal, call it from TypeScript via fetch.

Python Side

# inference.py
import modal

app = modal.App("my-inference-api")

image = modal.Image.debian_slim().uv_pip_install(["fastapi[standard]", "transformers", "torch"])

@app.function(image=image, gpu="A10G")
@modal.fastapi_endpoint(method="POST")
def predict(payload: dict):
    # GPU-accelerated inference
    text = payload["text"]
    result = run_model(text)
    return {"prediction": result}

TypeScript Side

const response = await fetch(MODAL_ENDPOINT, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(input),
  signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS), // Essential for cold starts
});

**Key requirements:** Named constant for URL (not hardcoded at call sites), `Content-Type: application/json` header (FastAPI rejects without it), `AbortSignal.timeout()` to handle cold start delays, typed request/response interfaces.

See [examples/core.md](examples/core.md) for a complete TypeScript client with error handling and typed interfaces.

---

Pattern 2: Authenticated Endpoints

Modal supports proxy auth tokens that protect endpoints without spinning up containers for unauthorized requests.

Python Side

@app.function(image=image, gpu="A10G")
@modal.fastapi_endpoint(method="POST", requires_proxy_auth=True)
def predict_secure(payload: dict):
    return {"prediction": run_model(payload["text"])}

TypeScript Side

headers: {
  "Content-Type": "application/json",
  "Modal-Key": process.env.MODAL_PROXY_KEY,     // Proxy auth token
  "Modal-Secret": process.env.MODAL_PROXY_SECRET,
},

**Why good:** Auth handled at Modal's proxy layer (no container spin-up for bad requests), env vars for credentials. Add explicit 401 handling in your error