Incident Response
Incidents are inevitable in any production system. The difference between a minor blip and a catastrophic failure is how prepared your team is to detect, respond, communicate, and learn. This guide covers the full incident lifecycle from classification through postmortem, with concrete templates and integration patterns.
Severity Classification
Every incident must be classified immediately. Severity determines response speed, communication cadence, and escalation paths.
| Severity | Impact | Examples | Response Time | Duration Target | |----------|--------|----------|---------------|-----------------| | SEV1 | Complete outage or data loss | Service down for all users, data corruption, security breach | 5 min | Mitigate < 1 hour | | SEV2 | Major degradation | Core feature broken for >10% users, payment failures | 15 min | Mitigate < 4 hours | | SEV3 | Minor degradation | Non-critical feature broken, elevated error rate | 1 hour | Resolve < 24 hours | | SEV4 | Cosmetic or low impact | UI glitch, misleading error message | Next business day | Resolve < 1 week |
Classification Decision Tree
Is the service completely unavailable to all users? -> SEV1
Is there a security breach or data exposure? -> SEV1
Is there data loss or corruption? -> SEV1
Is a core revenue feature broken for >10% users? -> SEV2
Is there financial impact (failed payments)? -> SEV2
Is a core feature broken for <10% users? -> SEV3
Is it a non-critical feature degradation? -> SEV3
Everything else -> SEV4On-Call Setup
| Component | Recommendation | Rationale | |-----------|---------------|-----------| | Rotation size | 5-8 engineers minimum | Allows 1-week shifts with recovery time | | Shift length | 1 week (Mon 09:00 to Mon 09:00) | Predictable, long enough for context | | Primary + Secondary | Always two people on call | Backup for escalation or unavailability | | Handoff meeting | 30 min at rotation start | Review active issues, recent changes, known risks | | Follow-the-sun | Split by timezone for global teams | No one wakes up at 3 AM regularly | | Compensation | On-call stipend + incident bonus | Recognizes the burden fairly |
PagerDuty Schedule and Escalation (Terraform)
resource "pagerduty_schedule" "primary_oncall" {
name = "Platform Primary On-Call"
time_zone = "America/New_York"
layer {
name = "Primary"
start = "2025-01-06T09:00:00-05:00"
rotation_virtual_start = "2025-01-06T09:00:00-05:00"
rotation_turn_length_seconds = 604800 # 1 week
users = [
pagerduty_user.alice.id, pagerduty_user.bob.id,
pagerduty_user.carol.id, pagerduty_user.dave.id,
pagerduty_user.eve.id,
]
}
}
resource "pagerduty_escalation_policy" "platform" {
name = "Platform Escalation"
num_loops = 2
rule {
escalation_delay_in_minutes = 5
target { type = "schedule_reference"; id = pagerduty_schedule.primary_oncall.id }
}
rule {
escalation_delay_in_minutes = 10
target { type = "schedule_reference"; id = pagerduty_schedule.secondary_oncall.id }
}
rule {
escalation_delay_in_minutes = 15
target { type = "user_reference"; id = pagerduty_user.engineering_manager.id }
}
}War Room Protocol
When a SEV1 or SEV2 is declared, open a war room -- a structured environment for incident resolution.
War Room Flow
1. DETECT Alert fires -> On-call acknowledges within 5 min
2. TRIAGE Classify severity. SEV1/SEV2 -> open war room
3. ASSEMBLE IC assigned. Slack: #inc-YYYYMMDD-description. Video bridge opened.
4. ROLES IC | Comms Lead | Operations Lead | Scribe | SMEs
5. INVESTIGATE What changed? What is blast radius? What do signals show?
6. MITIGATE Priority: restore service (rollback, feature flag, scale, failover)
7. RESOLVE Service stable. IC declares resolved.
8. FOLLOW-UP Postmortem within 48 hours. Action items tracked.Incident Channel Template
**INCIDENT DECLARED**
Severity: SEV1 | Title: Order processing failing for all users
Detected: 2025-03-15 14:32 UTC
Impact: All users unable to complete checkout
**ROLES** IC: @alice | Comms: @bob | Ops: @carol | Scribe: @dave
**LINKS**
Status Page: https://status.example.com
Runbook: https://wiki.internal/runbooks/order-processing
Dashboard: https://grafana.internal/d/orders-overview
**TIMELINE**
14:32 - Alert fired: HighErrorBudgetBurnRate_Fast
14:35 - On-call acknowledged, war room opened
14:38 - Identified: deploy changed payment gateway config
14:42 - Rollback initiated
14:47 - Rollback complete, error rate dropping
14:55 - Fully restored | 15:00 - ResolvedRunbook Template
Every alert should link to a runbook with diagnosis, mitigation, and recovery steps.
# runbooks/order-service-high-error-rate.yml
metadata:
title: "Order Service High Error Rate"
service: order-service
severity: SEV1/SEV2
owner: platform-team
alert_names: [HighErrorBudgetBurnRate_Fast, HighErrorBudgetBurnRate_Slow]
impact: "Users cannot checkout. Revenue impact ~$2,400/min at peak."
diagnosis:
- step: Check recent deployments
command: "kubectl -n production rollout history deployment/order-service"
expected: "If deploy correlates with error onset, proceed to rollback"
- step: Check error logs
command: '{service="order-service"} |= "error" | json | level="error"'
expected: "Identify error type: database, upstream, or application"
- step: Check downstream dependencies
command: "curl -s https://payment-service.internal/healthz | jq ."
expected: "All healthy. If not, see payment-service-down runbook"
- step: Check database performance
command: "psql -h orders-db -c \"SELECT pid, state, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start LIMIT 20;\""
expected: "No long-running queries or lock contention"
mitigation:
- option: Rollback last deployment
when: "Error onset correlates with a deployment"
command: "kubectl -n production rollout undo deployment/order-service"
verify: "Error rate returns to baseline within 5 minutes"
- option: Scale up
when: "Capacity-related errors (connection pool, CPU)"
command: "kubectl -n production scale deployment/order-service --replicas=10"
verify: "Connection pool usage drops below 80%"
- option: Circuit breaker
when: "Payment service is root cause"
command: "kubectl set env deployment/order-service PAYMENT_CIRCUIT_BREAKER=open"
verify: "5xx rate drops, orders queue for retry"
- option: Database failover
when: "Primary database unresponsive"
command: "aws rds failover-db-cluster --db-cluster-identifier orders-cluster"
verify: "Connections re-establish within 60 seconds"
recovery:
- "Confirm baseline error rate for 15 minutes"
- "Check for data inconsistencies from failed transactions"
- "Update status page to resolved"
- "Schedule postmortem within 48 hours"Escalation Policies
| Trigger | Action | Timeout | |---------|--------|---------| | Alert fires | Page primary on-call | -- | | No ack in 5 min | Escalate to secondary | 5 min | | No ack in 15 min | Escalate to engineering manager | 10 min | | SEV1 declared | Auto-notify VP Eng + CTO | Immediate | | 30 min without mitigation | IC requests additional responders | IC decision | | Customer data exposed | Notify Security + Legal | Immediate |
Communication Templates
# Status Page -- Investigating
**[Investigating] Elevated error rates on checkout**
We are investigating errors during checkout. Our team is engaged.
Update within 30 minutes.
# Status Page -- Identified
**[Identified] Checkout errors caused by payment config issue**
Root cause identified. Fix deploying now. Update in 15 minutes.
# Status Page -- Resolved
**[Resolved] Checkout errors resolved**
Configuration rolled back at 14:47 U