ai-infrastructure-modal

Modal Patterns

Quick Guide: Modal is a serverless GPU compute platform where you define Python functions with decorators and Modal handles containers, scaling, and GPU provisioning. TypeScript apps interact with Modal via HTTP endpoints (calling @modal.fastapi_endpoint or @modal.asgi_app functions) or the modal npm SDK (calling functions directly via gRPC). Define container images, secrets, and volumes as code -- no YAML config files. Use modal deploy for production, modal serve for dev.

<critical_requirements>

CRITICAL: Before Using This Skill

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST define Modal functions in Python -- the TypeScript SDK can call functions and manage resources but cannot define them)

(You MUST use @modal.fastapi_endpoint (not the old @modal.web_endpoint) for simple web endpoints -- renamed in Modal 1.0)

(You MUST use modal.Volume for model weight caching -- @modal.build is deprecated in Modal 1.0)

(You MUST never hardcode secrets in Modal code -- use modal.Secret.from_name() and access via os.environ)

(You MUST bind to 0.0.0.0 (not 127.0.0.1) when using @modal.web_server)

</critical_requirements>

Auto-detection: Modal, modal, modal.App, modal.Image, modal.Volume, modal.Secret, modal.gpu, modal.fastapi_endpoint, modal.asgi_app, modal.web_server, modal.Cron, modal.Period, modal deploy, modal serve, MODAL_TOKEN_ID, MODAL_TOKEN_SECRET, ModalClient

When to use:

• Deploying ML models (vLLM, Hugging Face, custom PyTorch) on serverless GPUs
• Creating HTTP API endpoints backed by GPU compute for TypeScript apps to consume
• Running scheduled GPU jobs (fine-tuning, batch inference, data processing)
• Calling Modal functions from TypeScript using the modal npm SDK
• Building AI inference pipelines with auto-scaling and pay-per-second billing

Key patterns covered:

• Web endpoints (@modal.fastapi_endpoint, @modal.asgi_app, @modal.web_server) for HTTP access
• TypeScript client patterns (fetch-based and modal npm SDK)
• Container images, secrets, volumes, and GPU configuration
• Model serving with vLLM and custom inference
• Scheduled functions and deployment

When NOT to use:

• Pure Python ML workloads with no TypeScript consumer -- this skill focuses on the TypeScript interaction surface
• Simple CPU-only tasks where a regular server or cloud function suffices
• When you need persistent long-running servers (Modal scales to zero by default)
• You need sub-100ms cold starts (Modal cold starts are 2-4 seconds)
• You need persistent WebSocket connections beyond request/response

Examples Index

• [Core: Web Endpoints & TypeScript Client](examples/core.md) -- Defining endpoints, calling from TypeScript, authentication, GPU functions, images, secrets, volumes
• [Quick API Reference](reference.md) -- CLI commands, decorator parameters, URL patterns, GPU types

<philosophy>

Philosophy

Modal eliminates infrastructure management for GPU workloads. Everything is code -- container images, GPU allocation, secrets, volumes, scaling rules. There are no YAML configs, Dockerfiles, or Kubernetes manifests.

Core principles:

1. Infrastructure as Python code -- Container images, GPU types, secrets, and volumes are all declared as Python decorators and objects. No separate config files.
2. Serverless GPU scaling -- Functions scale from zero to hundreds of GPUs automatically. You pay per second of compute, not for idle capacity.
3. Two interaction models for TypeScript -- Call Modal via HTTP endpoints (most common) or via the modal npm SDK for direct function invocation without HTTP overhead.
4. Immutable deployments -- modal deploy creates a named, persistent deployment with stable URLs. modal serve creates ephemeral dev endpoints.

</philosophy>

<patterns>

Core Patterns

Pattern 1: Web Endpoint (TypeScript Consumption)

The most common pattern: define a Python endpoint on Modal, call it from TypeScript via fetch.

Python Side

# inference.py
import modal

app = modal.App("my-inference-api")

image = modal.Image.debian_slim().uv_pip_install(["fastapi[standard]", "transformers", "torch"])

@app.function(image=image, gpu="A10G")
@modal.fastapi_endpoint(method="POST")
def predict(payload: dict):
    # GPU-accelerated inference
    text = payload["text"]
    result = run_model(text)
    return {"prediction": result}

TypeScript Side

const response = await fetch(MODAL_ENDPOINT, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(input),
  signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS), // Essential for cold starts
});

Key requirements: Named constant for URL (not hardcoded at call sites), Content-Type: application/json header (FastAPI rejects without it), AbortSignal.timeout() to handle cold start delays, typed request/response interfaces.

See [examples/core.md](examples/core.md) for a complete TypeScript client with error handling and typed interfaces.

Pattern 2: Authenticated Endpoints

Modal supports proxy auth tokens that protect endpoints without spinning up containers for unauthorized requests.

Python Side

@app.function(image=image, gpu="A10G")
@modal.fastapi_endpoint(method="POST", requires_proxy_auth=True)
def predict_secure(payload: dict):
    return {"prediction": run_model(payload["text"])}

TypeScript Side

headers: {
  "Content-Type": "application/json",
  "Modal-Key": process.env.MODAL_PROXY_KEY,     // Proxy auth token
  "Modal-Secret": process.env.MODAL_PROXY_SECRET,
},

Why good: Auth handled at Modal's proxy layer (no container spin-up for bad requests), env vars for credentials. Add explicit 401 handling in your error logic.

See [examples/core.md](examples/core.md) for a complete authenticated TypeScript client with error handling.

Pattern 3: Modal npm SDK (Direct Function Calls)

For TypeScript apps that need to call Modal functions without HTTP overhead. Requires Node 22+.

import { ModalClient } from "modal";
const modal = new ModalClient(); // Create once, reuse
const fn = await modal.functions.fromName("my-inference-api", "predict");
const result = await fn.remote([text]); // sync call
const call = await fn.spawn([text]); // async (fire-and-forget)
const later = await call.get(); // retrieve result later

Why good: No HTTP serialization overhead, typed SDK, supports async spawn for long-running jobs

See [examples/core.md](examples/core.md) for complete TypeScript SDK patterns including error handling and fire-and-forget job IDs.

When to use: Backend-to-Modal calls where you control the Node.js runtime (Node 22+). Not for browser or edge runtimes.

Pattern 4: GPU Functions and Container Images

Modal functions define their compute environment inline.

import modal

app = modal.App("gpu-inference")

# Container image with ML dependencies
inference_image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install(["torch==2.5.0", "transformers==4.47.0", "accelerate"])
    .apt_install(["libgl1"])
)

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

@app.function(
    image=inference_image,
    gpu="A100",                   # GPU type: "T4", "A10G", "A100", "H100", etc.
    secrets=[modal.Secret.from_name("huggingface-secret")],
    volumes={"/models": modal.Volume.from_name("model-cache", create_if_missing=True)},
    min_containers=1,             # Keep warm to avoid cold starts
    scaledown_window=300,         # Seconds before scaling to zero
)
def generate(prompt: str) -> str:
    from transformers import AutoModelForCausalLM, AutoTokenizer
    # Load from volume cache
    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, cache_dir="/models")
    mo