PagerDuty Incident Management

Overview

Incidents are the core resource in PagerDuty — created automatically from alert rules, event orchestrations, or manually by responders. They represent an active service disruption that requires acknowledgment and resolution. For MSPs, PagerDuty incidents typically represent internal infrastructure issues or customer-impacting events routed from monitoring integrations (Datadog, CloudWatch, Nagios, etc.).

PagerDuty has 14 MCP tools for incident management — more than any other domain.

MCP Tools

Core Incident Tools

Investigation Tools

Custom Fields

Key Concepts

Incident Lifecycle

┌───────────┐   Alert fires   ┌──────────────┐   Responder acts   ┌───────────┐
│ triggered │ ──────────────> │ acknowledged │ ──────────────────> │ resolved  │
└───────────┘                 └──────────────┘                     └───────────┘
      │
      │ Nobody acks within escalation timeout
      ▼
 [Escalates to next layer in escalation policy]

Urgency Levels

Priority

Priorities are account-configured (P1–P5 is common). Priority is separate from urgency:

• Urgency — controls notification escalation behavior
• Priority — business classification of impact severity

Alerts vs. Incidents

• Alert — Raw signal from a monitoring integration (many per incident)
• Incident — The grouped, actionable work item (one or more alerts)

list_incident_alerts shows you which monitoring signals triggered an incident.

Field Reference

Common Workflows

Triage Open Incidents

1. Call list_incidents with statuses[]=triggered&statuses[]=acknowledged, sorted by created_at desc
2. Group by urgency: high urgency first
3. For each critical incident, call get_incident for full details
4. Call list_incident_alerts to understand the triggering signal
5. Call list_past_incidents on the incident ID to find similar historical events
6. Add acknowledgment note via create_incident_note with your investigation steps

Investigate a Triggered Incident

1. Call get_incident to get full details (service, escalation policy, assignments)
2. Call list_incident_alerts to see what monitoring signals fired
3. Call list_past_incidents — this uses AI to find similar past incidents and their resolutions
4. Call list_incident_log_entries for the full audit trail of notifications and escalations
5. Acknowledge via update_incident with status=acknowledged

Bulk Resolve After Maintenance

After a maintenance window resolves multiple incidents:

1. Call list_incidents with statuses[]=triggered&statuses[]=acknowledged filtered to the affected service
2. Call manage_incidents with all incident IDs and status=resolved
3. Add a resolution note to each with create_incident_note

Cross-Vendor PSA Ticket Correlation

PagerDuty incidents often need to be tracked in a PSA for billing/SLA reporting:

1. When a high-urgency PagerDuty incident is created, create a corresponding PSA ticket
2. Store the PagerDuty incident number (#342) in the PSA ticket body for cross-reference
3. Map PagerDuty urgency → PSA priority: high → Critical/High, low → Medium/Low
4. When the PagerDuty incident resolves, update the PSA ticket with the resolution timestamp
5. Use list_incident_log_entries to extract total response time for SLA tracking

Merge Duplicate Incidents

When the same root cause fires multiple incidents:

1. Identify the primary incident (earliest or highest urgency)
2. Call merge_incidents with the primary incident id and the IDs of the duplicates in source_incidents[]
3. PagerDuty merges all alerts and log entries into the primary incident
4. The secondary incidents are automatically resolved

Error Handling

Best Practices

1. Use list_past_incidents — PagerDuty's AI similarity search often surfaces the exact runbook needed
2. Filter by service and team — Always scope incident queries to relevant services to avoid noise
3. Acknowledge before investigating — Stops escalation clock while you triage
4. Add notes as you go — create_incident_note builds a shared investigation timeline
5. Merge duplicates immediately — Reduces responder confusion during active incidents
6. Use since/until for reports — Unbounded queries on large accounts are slow

Related Skills

• On-Call Management — Who is on-call, escalation policies, overrides
• API Patterns — Token format, 66-tool reference, pagination