chaos-engineering

When this skill is activated, always start your first response with the 🧢 emoji.

Chaos Engineering

A practitioner's framework for running controlled failure experiments in production systems. This skill covers how to design, execute, and learn from chaos experiments - from simple latency injections to full game days - with an emphasis on safety, minimal blast radius, and translating findings into durable resilience improvements.

When to use this skill

Trigger this skill when the user:

• Wants to design a chaos experiment or fault injection scenario
• Is setting up a chaos engineering program from scratch
• Needs to implement network latency, packet loss, or service dependency failures
• Is planning or facilitating a game day exercise
• Needs to validate circuit breakers, retries, or failover logic under real failure conditions
• Wants to measure and improve MTTR (Mean Time to Recovery)
• Is evaluating chaos tooling (Chaos Monkey, Litmus, Gremlin, AWS Fault Injection Simulator)

Do NOT trigger this skill for:

• Writing standard retry or circuit breaker code without the intent to test it under chaos (use backend-engineering skill)
• Load testing or performance benchmarking that does not involve failure injection (use performance-engineering skill)

Key principles

1. Define steady state before breaking anything - You cannot detect a deviation

without a baseline. Before every experiment, define the precise metric (p99 latency, error rate, success count) that proves the system is healthy. If the system is already degraded, stop and fix it first.

1. Start small in staging, graduate to production slowly - Every experiment starts in

a non-production environment. Only move to production after the hypothesis is proven correct in staging and blast radius is understood. Even in production, target a small traffic percentage or a single availability zone first.

1. Minimize blast radius - The experiment scope must be as small as possible. Isolate

the failure to one service, one host, or one region. Have a kill switch ready before starting. The goal is learning, not causing an incident.

1. Build the hypothesis before turning on the failure - A hypothesis has three parts:

"When X happens, the system will Y, as evidenced by Z metric." Without a pre-written hypothesis you cannot distinguish a passing experiment from an outage.

1. Automate experiments and run them continuously - A chaos experiment run once is a

one-time curiosity. Automated experiments that run on every deploy catch regressions before production. The goal is a resilience gate in CI/CD, not a quarterly fire drill.

Core concepts

Steady State Hypothesis

The foundation of every experiment. A steady state is a measurable, normal behavior of the system:

Hypothesis template:
  "Under normal conditions, [service] processes [metric] at [baseline value].
   When [failure condition] is introduced, [metric] will remain within [acceptable range]
   because [resilience mechanism] will compensate."

Example:
  "Under normal conditions, the checkout service processes 95% of requests in <500ms.
   When the inventory service has 500ms of added latency, checkout p99 will remain
   <800ms because the circuit breaker will open and return cached availability data."

Metrics for steady state (RED method):

• Rate - requests per second
• Errors - error rate (%)
• Duration - latency percentiles (p50, p95, p99)

Blast Radius

The maximum potential impact of the experiment if something goes wrong. Always quantify before starting:

| Blast radius dimension | Example | How to constrain | |---|---|---| | Traffic percentage | 5% of prod requests | Feature flags, canary routing | | Infrastructure scope | 1 of 3 availability zones | Target specific AZ tags | | Service scope | One pod/instance in the fleet | Target single hostname | | Time scope | 10-minute window | Automated kill switch with timeout |

Experiment Lifecycle

1. DEFINE    -> Write steady state hypothesis + success/failure criteria
2. SCOPE     -> Identify target environment, blast radius, and rollback mechanism
3. INSTRUMENT -> Confirm observability is in place to measure the hypothesis metric
4. RUN       -> Inject failure; observe metric in real time
5. ANALYZE   -> Did steady state hold? If not, why? What was the real failure mode?
6. IMPROVE   -> Fix the gap. Update runbooks. Automate the experiment.
7. REPEAT    -> Re-run to confirm the fix. Graduate to broader scope.

Failure Modes Taxonomy

| Category | Examples | Common tools | |---|---|---| | Network | Latency, packet loss, DNS failure, partition | tc netem, Toxiproxy, Gremlin | | Resource | CPU saturation, memory pressure, disk full, fd exhaustion | stress-ng, Chaos Monkey | | Dependency | Service unavailable, slow response, bad responses (500/400) | Wiremock, Litmus, FIS | | Infrastructure | Pod kill, node drain, AZ outage, region failover | Chaos Monkey, Litmus, FIS | | Application | Exception injection, clock skew, thread pool exhaustion | Byte Monkey, custom middleware | | Data | Corrupt payload, missing field, schema mismatch | Custom fuzz harness |

Common tasks

Design a chaos experiment

Use this template to structure every experiment:

## Chaos Experiment: [Short Name]

**Date:** YYYY-MM-DD
**Hypothesis:**
  When [failure condition], [service] will [expected behavior]
  as evidenced by [metric staying within range].

**Steady State (before):**
  - Metric: checkout.success_rate
  - Baseline: >= 99.5%
  - Measured via: Datadog SLO dashboard / Prometheus query

**Failure injection:**
  - Tool: Toxiproxy / Litmus / AWS FIS
  - Target: inventory-service, 1 of 5 pods
  - Type: HTTP 503 response, 100% of requests to /api/stock
  - Duration: 10 minutes

**Blast radius:**
  - Scope: Single pod in staging environment
  - Traffic affected: ~20% of inventory requests
  - Kill switch: `kubectl delete chaosexperiment inventory-latency`

**Success criteria:**
  - checkout.success_rate remains >= 99.5% during injection
  - Circuit breaker opens within 30s
  - Fallback (cached stock) is served to users

**Failure criteria:**
  - checkout.success_rate drops below 99% for > 2 minutes
  - Any user-visible 500 errors during injection

**Result:** [PASS / FAIL]
**Finding:** [What actually happened]
**Action:** [Ticket number + fix description]

Implement network latency injection

Inject latency at the network level using Linux Traffic Control (tc) or Toxiproxy (application-level proxy). Prefer Toxiproxy for service-specific targeting; prefer tc for host-level experiments.

Using Toxiproxy (service-level, recommended for staging):

# Install and start Toxiproxy
toxiproxy-server &

# Create a proxy for the downstream service
toxiproxy-cli create --listen 0.0.0.0:8474 --upstream inventory-svc:8080 inventory_proxy

# Add 200ms of latency with 50ms jitter to 100% of connections
toxiproxy-cli toxic add inventory_proxy \
  --type latency \
  --attribute latency=200 \
  --attribute jitter=50 \
  --toxicity 1.0

# Point your service at localhost:8474 instead of inventory-svc:8080
# ... run the experiment, observe metrics ...

# Remove the toxic (kill switch)
toxiproxy-cli toxic remove inventory_proxy --toxicName latency_downstream

Using tc netem (host-level, for infrastructure experiments):

# Add 300ms latency + 30ms jitter to all outbound traffic on eth0
sudo tc qdisc add dev eth0 root netem delay 300ms 30ms

# Add 10% packet loss
sudo tc qdisc change dev eth0 root netem loss 10%

# Remove (kill switch)
sudo tc qdisc del dev eth0 root

Always test the kill switch before starting the experiment. A failed kill switch

turns a chaos experiment into a real incident.

Simulate service dependency failure

Test what happens when a downstream service becomes unavailable. Use Wiremock or a simple m