/ai-infrastructure-replicate

name: ai-infrastructure-replicate
description: Replicate SDK patterns for TypeScript/Node.js -- client setup, predictions, streaming, webhooks, file handling, model versioning, deployments, and training

Replicate SDK Patterns

> **Quick Guide:** Use the `replicate` npm package to run open-source ML models on serverless GPUs. Use `replicate.run()` for synchronous execution that returns output directly, `replicate.stream()` for SSE-based streaming, or `replicate.predictions.create()` for async background jobs with webhook notifications. Models are referenced as `owner/model` (uses latest version) or `owner/model:version` (pinned). File outputs are `FileOutput` objects implementing `ReadableStream`. Cold starts are expected for infrequently-used models -- use deployments with `min_instances` to keep models warm.

---

<critical_requirements>

CRITICAL: Before Using This Skill

**(You MUST never hardcode API tokens -- always use environment variables via `process.env.REPLICATE_API_TOKEN`)**

**(You MUST handle `FileOutput` objects for models that return files -- do not assume outputs are plain strings or URLs)**

**(You MUST validate webhooks using `validateWebhook()` from the `replicate` package -- never trust unverified webhook payloads)**

**(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)**

**(You MUST specify model versions (`owner/model:version`) in production to ensure reproducible results -- unversioned references use the latest, which can change)**

</critical_requirements>

---

**Auto-detection:** Replicate, replicate, replicate.run, replicate.stream, replicate.predictions, replicate.deployments, replicate.trainings, replicate.models, FileOutput, validateWebhook, REPLICATE_API_TOKEN, serverless GPU, cold start, webhook_events_filter

**When to use:**

• Running open-source ML models (Llama, Stable Diffusion, Whisper, etc.) without managing GPU infrastructure
• Generating images, transcribing audio, running LLMs, or any ML inference via API
• Streaming LLM output in real-time with server-sent events
• Processing predictions asynchronously with webhook notifications
• Fine-tuning models with custom training data
• Running models on dedicated hardware with custom scaling via deployments

**Key patterns covered:**

• Client initialization and configuration (auth, user agent, file encoding)
• Running predictions (`replicate.run()`, `replicate.predictions.create()`, `replicate.wait()`)
• Streaming output (`replicate.stream()` with SSE events)
• Model versioning (`owner/model` vs `owner/model:version`)
• File input/output handling (`FileOutput`, file uploads, `Buffer` inputs)
• Webhooks (setup, event filtering, signature validation)
• Deployments (custom hardware, scaling, keeping models warm)
• Training / fine-tuning

**When NOT to use:**

• You need a unified multi-provider LLM SDK (OpenAI, Anthropic, Google) -- use a provider-agnostic SDK
• You want to run models locally -- Replicate is a cloud-only serverless platform
• You need sub-second latency guarantees without deployments -- cold starts can take minutes

---

Examples Index

• [Core: Setup, Predictions & Files](examples/core.md) -- Client init, run(), predictions.create(), wait(), file I/O, error handling
• [Streaming & Webhooks](examples/streaming-webhooks.md) -- stream(), SSE events, webhook setup, signature validation
• [Deployments & Training](examples/deployments-training.md) -- Custom hardware, scaling, fine-tuning, model management
• [Quick API Reference](reference.md) -- Method signatures, constructor options, error types, model reference format

---

<philosophy>

Philosophy

Replicate provides **serverless GPU infrastructure** for running open-source ML models. You send inputs, Replicate allocates GPU hardware, runs the model, and returns outputs. No Docker, no CUDA drivers, no GPU provisioning.

**Core principles:**

1. **Serverless execution** -- Models run on-demand on Replicate's infrastructure. You pay only for compute time. Cold starts are a trade-off for not maintaining always-on GPUs. 2. **Model marketplace** -- Thousands of community and official models available at `replicate.com/explore`. Run any public model with just its identifier. 3. **Version pinning for reproducibility** -- Models are versioned with SHA-256 hashes. Pin to a version in production (`owner/model:abc123...`) to guarantee identical behavior across deploys. 4. **Three execution modes** -- `replicate.run()` for synchronous wait, `replicate.stream()` for real-time SSE output, `replicate.predictions.create()` for fire-and-forget with webhooks. 5. **File-first I/O** -- Many models accept and produce files (images, audio, video). The SDK handles file uploads automatically and returns `FileOutput` objects for file outputs.

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Client Setup

Initialize the Replicate client. It auto-reads `REPLICATE_API_TOKEN` from the environment.

// lib/replicate.ts -- basic setup
import Replicate from "replicate";

const replicate = new Replicate();

export { replicate };

// lib/replicate.ts -- explicit auth + custom user agent
import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted
  userAgent: "my-app/1.0.0",
});

export { replicate };

**Why good:** Minimal setup, env var auto-detected, explicit auth optional but useful for clarity

// BAD: Hardcoded token
const replicate = new Replicate({
  auth: "r8_abc123...",
});

**Why bad:** Hardcoded API token is a security risk, will leak in version control

**See:** [examples/core.md](examples/core.md) for full constructor options, error handling patterns

---

Pattern 2: Running Predictions

Use `replicate.run()` for synchronous execution. Returns the model output directly.

// Run an image generat