sre-patterns

SRE Patterns

Best practices for building and operating reliable systems using Site Reliability Engineering principles.

SLO / SLI / SLA Definitions

These three concepts form the foundation of SRE. They are distinct and frequently confused.

| Concept | Definition | Owner | Example | |---------|-----------|-------|---------| | SLI (Service Level Indicator) | A quantitative measurement of a service attribute | Engineering | 99.2% of requests completed in < 300ms | | SLO (Service Level Objective) | A target value or range for an SLI | Engineering + Product | 99.5% of requests must complete in < 300ms | | SLA (Service Level Agreement) | A contract with consequences for missing an SLO | Business + Legal | 99.9% uptime or customer receives service credits |

Relationship

SLI (what you measure)
 --> SLO (what you target, always stricter than SLA)
  --> SLA (what you promise externally, with penalties)

Key rule: SLO must be stricter than SLA. If your SLA promises 99.9% uptime, your internal SLO should target 99.95%. The gap is your safety margin.

SLI Specification

SLIs must be precise, measurable, and tied to user experience. Vague indicators lead to meaningless objectives.

SLI Types by Service Category

| Service Type | SLI Category | Good Event | Valid Event | |-------------|-------------|------------|-------------| | Request-driven | Availability | Response status < 500 | All HTTP requests | | Request-driven | Latency | Response time < 300ms | All HTTP requests | | Data pipeline | Freshness | Data age < 10 minutes | All data records | | Data pipeline | Correctness | Records with no processing errors | All processed records | | Storage system | Durability | Objects retrievable after write | All stored objects | | Storage system | Throughput | Read operations < 50ms | All read operations |

SLI Specification Example

# sli-specification.yaml
service: payment-api
slis:
  - name: availability
    description: Proportion of successful requests
    specification:
      good_event: "HTTP response status code is not 5xx"
      valid_event: "All HTTP requests to /api/v1/payments/*"
      measurement_source: load_balancer_logs
      measurement_window: rolling_28_days
    implementation:
      numerator: "count(status < 500)"
      denominator: "count(all requests)"
      exclude:
        - health_check_endpoints
        - synthetic_monitoring_requests

  - name: latency
    description: Proportion of requests served within threshold
    specification:
      good_event: "HTTP response completes within 300ms"
      valid_event: "All non-background HTTP requests"
      measurement_source: server_side_metrics
      measurement_window: rolling_28_days
    implementation:
      numerator: "count(duration_ms <= 300)"
      denominator: "count(all requests)"
      percentile_targets:
        p50: 100ms
        p95: 250ms
        p99: 500ms

Error Budget Calculation

The error budget is the inverse of your SLO -- the amount of unreliability you can tolerate.

Formula

Error Budget = 1 - SLO target

Example:
  SLO = 99.9% availability
  Error Budget = 1 - 0.999 = 0.1%

  In a 30-day month (43,200 minutes):
  Budget = 43,200 * 0.001 = 43.2 minutes of downtime allowed

Error Budget by SLO Level

| SLO Target | Error Budget (30 days) | Error Budget (per quarter) | Practical Meaning | |-----------|----------------------|--------------------------|-------------------| | 99.0% | 7 hours 12 min | 21 hours 36 min | Tolerates significant outages | | 99.5% | 3 hours 36 min | 10 hours 48 min | Weekly maintenance window feasible | | 99.9% | 43 minutes 12 sec | 2 hours 9 min | No room for long outages | | 99.95% | 21 minutes 36 sec | 1 hour 5 min | Requires high automation | | 99.99% | 4 minutes 19 sec | 12 min 58 sec | Requires redundancy at every layer |

Error Budget Burn Rate

Burn rate measures how fast you are consuming your error budget relative to the budget period.

# error_budget_tracking.py

def calculate_burn_rate(
    error_count: int,
    total_requests: int,
    slo_target: float,
    budget_period_hours: float,
    elapsed_hours: float
) -> dict:
    """Calculate error budget consumption and burn rate."""
    error_budget = 1.0 - slo_target
    current_error_rate = error_count / total_requests if total_requests > 0 else 0
    budget_consumed = current_error_rate / error_budget if error_budget > 0 else float('inf')

    # Burn rate: 1.0 means consuming budget exactly on pace
    # > 1.0 means burning faster than sustainable
    expected_consumed = elapsed_hours / budget_period_hours
    burn_rate = budget_consumed / expected_consumed if expected_consumed > 0 else 0

    remaining_budget = max(0, error_budget - current_error_rate)
    hours_remaining = (
        (remaining_budget / current_error_rate) * elapsed_hours
        if current_error_rate > 0 else float('inf')
    )

    return {
        "slo_target": slo_target,
        "error_budget_total": error_budget,
        "current_error_rate": current_error_rate,
        "budget_consumed_pct": budget_consumed * 100,
        "burn_rate": burn_rate,
        "hours_until_exhausted": hours_remaining,
    }

# Example usage:
# 99.9% SLO, 720-hour budget period (30 days), 168 hours elapsed (1 week)
result = calculate_burn_rate(
    error_count=150,
    total_requests=500_000,
    slo_target=0.999,
    budget_period_hours=720,
    elapsed_hours=168
)
# burn_rate > 1.0 --> alerting threshold

Error Budget Policy

An error budget policy defines what happens when the budget is consumed. Without a policy, the budget is just a number.

# error-budget-policy.yaml
service: payment-api
slo: 99.9% availability (rolling 28 days)
policy_owner: payments-team-lead
approved_by: vp-engineering
effective_date: 2025-01-15

budget_thresholds:
  - level: normal
    condition: "budget_consumed < 50%"
    actions:
      - Continue normal feature development
      - Standard deployment cadence (daily)
      - Routine reliability improvements as scheduled

  - level: caution
    condition: "budget_consumed >= 50% AND < 75%"
    actions:
      - Review recent deployments for reliability impact
      - Increase monitoring alert sensitivity
      - Prioritize known reliability-related bugs
      - Reduce deployment frequency to twice per week

  - level: critical
    condition: "budget_consumed >= 75% AND < 100%"
    actions:
      - Halt all non-reliability feature work
      - Require SRE approval for every deployment
      - Deploy only bug fixes and reliability improvements
      - Conduct focused reliability review within 48 hours
      - Notify stakeholders of SLO risk

  - level: exhausted
    condition: "budget_consumed >= 100%"
    actions:
      - Feature freeze until budget replenishes or root cause resolved
      - Emergency reliability review within 24 hours
      - Postmortem required for each new incident
      - All deployments require SRE sign-off and canary phase
      - Weekly status report to VP Engineering

escalation:
  budget_dispute: "Escalate to VP Engineering for arbitration"
  exemptions: "Launch exemptions require VP+ approval with risk acceptance"

Toil Measurement and Reduction

Toil is work that is manual, repetitive, automatable, reactive, and scales linearly with service growth. Toil is the enemy of reliability engineering.

Toil Characteristics

| Characteristic | Description | Example | |---------------|-------------|---------| | Manual | Requires a human to perform | SSH into server to restart service | | Repetitive | Done more than once or twice | Weekly cert rotation by hand | | Automatable | A machine could do it | Copying logs to analysis bucket | | Reactive | Triggered by an event, not proactive | Responding to disk full alerts | | No enduring value | Does not improve the service | Re-running failed batch jobs | | Scales with service | More instances = more work | Manually updating configs per