runbook-generator

Runbook Generator

Tier: POWERFUL Category: Engineering / SRE Maintainer: Claude Skills Team

Overview

Analyze a codebase and generate production-grade operational runbooks with copy-paste commands, verification checks after every step, rollback procedures for every destructive action, escalation paths with contact information, and time estimates for capacity planning. Detects the stack (CI/CD, database, hosting, containers) and produces runbooks tailored to the actual infrastructure. Includes staleness detection to flag runbooks when referenced config files change.

Keywords

runbook, operational procedures, incident response, deployment, rollback, database maintenance, scaling, monitoring, on-call, SRE, postmortem

Core Capabilities

1. Stack Detection

• Identify CI/CD platform, database, hosting, and orchestration from repo files
• Map detected stack to appropriate runbook templates
• Extract connection strings, deployment commands, and infrastructure details

2. Runbook Types

• Deployment: pre-checks, deploy steps, smoke tests, rollback
• Incident response: triage, diagnose, mitigate, resolve, postmortem
• Database maintenance: backup, migration, vacuum, reindex
• Scaling: horizontal and vertical scaling procedures
• Monitoring: alert setup, dashboard configuration, on-call rotation

3. Format Discipline

• Numbered steps with copy-paste commands
• Verification check after EVERY step
• Time estimates for capacity planning
• Rollback procedure for every destructive action
• Escalation paths with decision criteria

4. Maintenance

• Staleness detection linked to config file modification dates
• Quarterly review cadence
• Staging dry-run validation framework

When to Use

• Codebase has no runbooks and you need to bootstrap them
• Existing runbooks are outdated or incomplete
• Onboarding a new engineer for on-call rotation
• Preparing for an incident response drill
• Post-incident improvement: updating runbooks with lessons learned

Stack Detection

Scan the repository before writing any runbook:

# CI/CD Platform
[ -d ".github/workflows" ] && echo "GitHub Actions"
[ -f ".gitlab-ci.yml" ]    && echo "GitLab CI"
[ -f "Jenkinsfile" ]       && echo "Jenkins"

# Database
grep -rl "postgres\|postgresql" package.json pyproject.toml 2>/dev/null && echo "PostgreSQL"
grep -rl "mysql\|mariadb" package.json 2>/dev/null && echo "MySQL"
grep -rl "mongodb\|mongoose" package.json 2>/dev/null && echo "MongoDB"

# Hosting
[ -f "vercel.json" ]       && echo "Vercel"
[ -f "fly.toml" ]          && echo "Fly.io"
[ -f "railway.toml" ]      && echo "Railway"
[ -d "terraform" ]         && echo "Terraform (custom cloud)"
[ -d "k8s" ] || [ -d "kubernetes" ] && echo "Kubernetes"
[ -f "docker-compose.yml" ] && echo "Docker Compose"

# Framework
[ -f "next.config.mjs" ] || [ -f "next.config.ts" ] && echo "Next.js"
grep -q "fastapi" requirements.txt 2>/dev/null && echo "FastAPI"
[ -f "go.mod" ] && echo "Go"

Deployment Runbook Template

# Deployment Runbook — [App Name]

**Stack:** [Framework] + [Database] + [Hosting]
**Last verified:** YYYY-MM-DD
**Owner:** [Team Name]
**Estimated total time:** 15-25 minutes

---

## Staleness Check

| Config File | Last Modified | Affects Steps |
|-------------|--------------|---------------|
| vercel.json | `git log -1 --format=%ci -- vercel.json` | Deploy, Rollback |
| db/schema.ts | `git log -1 --format=%ci -- db/schema.ts` | Migration |
| .github/workflows/deploy.yml | `git log -1 --format=%ci -- .github/workflows/deploy.yml` | CI |

If any config was modified after "Last verified" date, review affected steps.

---

## Pre-Deployment Checklist
- [ ] All PRs merged to main
- [ ] CI passing on main branch
- [ ] Database migrations tested in staging
- [ ] Rollback plan confirmed
- [ ] On-call engineer notified

## Step 1: Verify CI Status (2 min)

Check latest CI run

gh run list --branch main --limit 3

Verify specific run

gh run view <run-id>

VERIFY: Latest run shows green checkmark. If red, do not proceed.

## Step 2: Apply Database Migrations (5 min)

Staging first

DATABASE_URL=$STAGING_DB_URL pnpm db:migrate

Verify migration applied

DATABASE_URL=$STAGING_DB_URL pnpm db:migrate status

VERIFY: Output shows "All migrations applied" with today's date.

Production (only after staging verification)

DATABASE_URL=$PROD_DB_URL pnpm db:migrate

VERIFY: Same output as staging. If error, see Rollback section.

WARNING: For migrations on tables with >1M rows, schedule during low-traffic window and monitor lock wait times.

## Step 3: Deploy to Production (5 min)

Option A: Git push triggers deployment

git push origin main

Option B: Manual trigger

vercel --prod

or: fly deploy

or: kubectl apply -f k8s/deployment.yaml

VERIFY: Deployment dashboard shows new version in progress. Note the deployment URL/ID for rollback.

## Step 4: Smoke Test (5 min)

Health check

curl -sf https://myapp.com/api/health | jq .

Critical user path

curl -sf https://myapp.com/api/v1/me \ -H "Authorization: Bearer $TEST_TOKEN" | jq '.id'

Check error rate (wait 2 minutes for data)

Dashboard: [link to monitoring dashboard]

VERIFY:
- Health returns `{"status": "ok", "db": "connected"}`
- User endpoint returns a valid user ID
- Error rate < 1% on monitoring dashboard

## Step 5: Monitor (10 min)

Watch these metrics for 10 minutes after deployment:
- Error rate: < 1% (dashboard: [link])
- P95 latency: < 200ms (dashboard: [link])
- Active DB connections: < 80% of max (query below)

psql $PROD_DB_URL -c "SELECT count(*) FROM pg_stat_activity;"

VERIFY: All metrics within normal range. If any spike, proceed to Rollback.

---

## Rollback

If smoke tests fail or metrics degrade:

Instant rollback via Vercel

vercel rollback [previous-deployment-url]

or Fly.io

fly releases --app myapp fly deploy --image [previous-image]

or Kubernetes

kubectl rollout undo deployment/myapp

Database rollback (ONLY if migration was applied in this deploy)

DATABASE_URL=$PROD_DB_URL pnpm db:rollback

VERIFY: Previous version serving traffic. Run smoke tests again.

---

## Escalation

| Level | Who | When | Contact |
|-------|-----|------|---------|
| L1 | On-call engineer | First responder | PagerDuty rotation |
| L2 | Platform lead | DB issues, rollback failures | Slack: @platform-lead |
| L3 | VP Engineering | Production down > 30 min | Phone: [number] |

Incident Response Runbook Template

# Incident Response Runbook

**Severity:** P1 (down), P2 (degraded), P3 (minor)
**Estimated time:** P1: 30-60 min, P2: 1-4 hours, P3: next business day

---

## Phase 1: Triage (5 min)

### Confirm the Incident

Is the app responding?

curl -sw "%{http_code}" https://myapp.com/api/health -o /dev/null

Check for errors in recent logs

vercel logs --since=15m | grep -i "error\|exception\|5[0-9][0-9]"

or: kubectl logs -l app=myapp --since=15m | grep -i error

VERIFY: 200 = app is up. 5xx or timeout = incident confirmed.

### Declare Severity

| Condition | Severity | Action |
|-----------|----------|--------|
| Site completely unreachable | P1 | Page L2/L3 immediately |
| Partial degradation or slow | P2 | Notify team channel |
| Single feature broken | P3 | Create ticket, fix in business hours |

### Communicate

Post to incident channel (adjust for your tool)

Slack: #incidents "INCIDENT: [severity] — [brief description]. Investigating. Updates every 15 min."

## Phase 2: Diagnose (10-15 min)

### Check Recent Changes

Was something just deployed?

vercel ls --limit 5

or: kubectl rollout history deployment/myapp

Recent commits

git log --oneline -10

### Check Database

Active queries (look for long-running or blocked queries)

psql $PROD_DB_URL -c " SELECT pid, now() - query_start AS duration, state, query FROM pg_stat_activity WHERE state != 'idle' ORDER