ai-infrastructure-replicate

Replicate SDK Patterns

Quick Guide: Use the replicate npm package to run open-source ML models on serverless GPUs. Use replicate.run() for synchronous execution that returns output directly, replicate.stream() for SSE-based streaming, or replicate.predictions.create() for async background jobs with webhook notifications. Models are referenced as owner/model (uses latest version) or owner/model:version (pinned). File outputs are FileOutput objects implementing ReadableStream. Cold starts are expected for infrequently-used models -- use deployments with min_instances to keep models warm.

<critical_requirements>

CRITICAL: Before Using This Skill

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST never hardcode API tokens -- always use environment variables via process.env.REPLICATE_API_TOKEN)

(You MUST handle FileOutput objects for models that return files -- do not assume outputs are plain strings or URLs)

(You MUST validate webhooks using validateWebhook() from the replicate package -- never trust unverified webhook payloads)

(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)

(You MUST specify model versions (owner/model:version) in production to ensure reproducible results -- unversioned references use the latest, which can change)

</critical_requirements>

Auto-detection: Replicate, replicate, replicate.run, replicate.stream, replicate.predictions, replicate.deployments, replicate.trainings, replicate.models, FileOutput, validateWebhook, REPLICATE_API_TOKEN, serverless GPU, cold start, webhook_events_filter

When to use:

• Running open-source ML models (Llama, Stable Diffusion, Whisper, etc.) without managing GPU infrastructure
• Generating images, transcribing audio, running LLMs, or any ML inference via API
• Streaming LLM output in real-time with server-sent events
• Processing predictions asynchronously with webhook notifications
• Fine-tuning models with custom training data
• Running models on dedicated hardware with custom scaling via deployments

Key patterns covered:

• Client initialization and configuration (auth, user agent, file encoding)
• Running predictions (replicate.run(), replicate.predictions.create(), replicate.wait())
• Streaming output (replicate.stream() with SSE events)
• Model versioning (owner/model vs owner/model:version)
• File input/output handling (FileOutput, file uploads, Buffer inputs)
• Webhooks (setup, event filtering, signature validation)
• Deployments (custom hardware, scaling, keeping models warm)
• Training / fine-tuning

When NOT to use:

• You need a unified multi-provider LLM SDK (OpenAI, Anthropic, Google) -- use a provider-agnostic SDK
• You want to run models locally -- Replicate is a cloud-only serverless platform
• You need sub-second latency guarantees without deployments -- cold starts can take minutes

Examples Index

• [Core: Setup, Predictions & Files](examples/core.md) -- Client init, run(), predictions.create(), wait(), file I/O, error handling
• [Streaming & Webhooks](examples/streaming-webhooks.md) -- stream(), SSE events, webhook setup, signature validation
• [Deployments & Training](examples/deployments-training.md) -- Custom hardware, scaling, fine-tuning, model management
• [Quick API Reference](reference.md) -- Method signatures, constructor options, error types, model reference format

<philosophy>

Philosophy

Replicate provides serverless GPU infrastructure for running open-source ML models. You send inputs, Replicate allocates GPU hardware, runs the model, and returns outputs. No Docker, no CUDA drivers, no GPU provisioning.

Core principles:

1. Serverless execution -- Models run on-demand on Replicate's infrastructure. You pay only for compute time. Cold starts are a trade-off for not maintaining always-on GPUs.
2. Model marketplace -- Thousands of community and official models available at replicate.com/explore. Run any public model with just its identifier.
3. Version pinning for reproducibility -- Models are versioned with SHA-256 hashes. Pin to a version in production (owner/model:abc123...) to guarantee identical behavior across deploys.
4. Three execution modes -- replicate.run() for synchronous wait, replicate.stream() for real-time SSE output, replicate.predictions.create() for fire-and-forget with webhooks.
5. File-first I/O -- Many models accept and produce files (images, audio, video). The SDK handles file uploads automatically and returns FileOutput objects for file outputs.

</philosophy>

<patterns>

Core Patterns

Pattern 1: Client Setup

Initialize the Replicate client. It auto-reads REPLICATE_API_TOKEN from the environment.

// lib/replicate.ts -- basic setup
import Replicate from "replicate";

const replicate = new Replicate();

export { replicate };

// lib/replicate.ts -- explicit auth + custom user agent
import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted
  userAgent: "my-app/1.0.0",
});

export { replicate };

Why good: Minimal setup, env var auto-detected, explicit auth optional but useful for clarity

// BAD: Hardcoded token
const replicate = new Replicate({
  auth: "r8_abc123...",
});

Why bad: Hardcoded API token is a security risk, will leak in version control

See: [examples/core.md](examples/core.md) for full constructor options, error handling patterns

Pattern 2: Running Predictions

Use replicate.run() for synchronous execution. Returns the model output directly.

// Run an image generation model
const [output] = await replicate.run("black-forest-labs/flux-schnell", {
  input: {
    prompt: "a serene mountain landscape at sunset",
  },
});

// output is a FileOutput object for image models
console.log(output.url()); // URL of generated image

// Run an LLM -- output is a string for text models
const output = await replicate.run("meta/meta-llama-3-70b-instruct", {
  input: {
    prompt: "Explain TypeScript generics in 3 sentences.",
    max_tokens: 512,
  },
});

console.log(output); // Text response

Why good: Simple API, returns output directly, destructuring works for array outputs (images)

// BAD: Not pinning version in production
const output = await replicate.run("community-user/experimental-model", {
  input: { prompt: "hello" },
});

Why bad: Community models without version pinning can change behavior unexpectedly when authors push updates

See: [examples/core.md](examples/core.md) for version pinning, predictions.create() + wait(), and progress callbacks

Pattern 3: Streaming

Use replicate.stream() for real-time SSE output from language models.

const stream = replicate.stream("meta/meta-llama-3-70b-instruct", {
  input: {
    prompt: "Write a short poem about TypeScript.",
    max_tokens: 512,
  },
});

for await (const event of stream) {
  if (event.event === "output") {
    process.stdout.write(event.data);
  }
}

Why good: Progressive output for better UX, event-based with typed event and data fields

// BAD: Using replicate.run() for user-facing LLM output
const output = await replicate.run("meta/meta-llama-3-70b-instruct", {
  input: { prompt: "Write a long essay..." },
});
// User waits for entire generation to complete before seeing anything

Why bad: No progressive feedback, user sees a blank screen for seconds

See: [examples/streaming-webhooks.md](examples/streaming-webhooks.md) for event types, error handling, cancellation

Pattern 4: Model Versioning

Models are reference