PagerDuty Incidents

PagerDuty Incident Management

Overview

Incidents are the core resource in PagerDuty — created automatically from alert rules, event orchestrations, or manually by responders. They represent an active service disruption that requires acknowledgment and resolution. For MSPs, PagerDuty incidents typically represent internal infrastructure issues or customer-impacting events routed from monitoring integrations (Datadog, CloudWatch, Nagios, etc.).

PagerDuty has 14 MCP tools for incident management — more than any other domain.

MCP Tools

Core Incident Tools

| Tool | Description | Key Parameters | |------|-------------|----------------| | list_incidents | List incidents with filters | statuses[], urgencies[], service_ids[], team_ids[], since, until | | get_incident | Get full incident details | id (required) | | create_incident | Create a new incident | title, service.id, urgency, body.details | | update_incident | Update incident fields | id, status, priority, urgency, title, assignments[] | | merge_incidents | Merge duplicate incidents | id (target), source_incidents[] | | snooze_incident | Snooze for a duration | id, duration (seconds) | | manage_incidents | Bulk update multiple incidents | incidents[] with individual update payloads |

Investigation Tools

| Tool | Description | Key Parameters | |------|-------------|----------------| | list_incident_alerts | Alerts that triggered this incident | id | | list_incident_notes | Comments and notes on the incident | id | | create_incident_note | Add a note to an incident | id, note.content | | list_incident_log_entries | Full audit trail of events | id, include[] | | list_past_incidents | Find historically similar incidents | id (finds similar to this incident) |

Custom Fields

| Tool | Description | |------|-------------| | get_incident_field_values | Read custom field values | | set_incident_field_values | Write custom field values |

Key Concepts

Incident Lifecycle

┌───────────┐   Alert fires   ┌──────────────┐   Responder acts   ┌───────────┐
│ triggered │ ──────────────> │ acknowledged │ ──────────────────> │ resolved  │
└───────────┘                 └──────────────┘                     └───────────┘
      │
      │ Nobody acks within escalation timeout
      ▼
 [Escalates to next layer in escalation policy]

| Status | Description | |--------|-------------| | triggered | Alert fired; awaiting acknowledgment | | acknowledged | Responder is working on it; escalation paused | | resolved | Incident is over; service restored |

Urgency Levels

| Urgency | Description | Notification Behavior | |---------|-------------|----------------------| | high | Critical service impact; requires immediate response | Phone, SMS, push | | low | Non-critical; informational or degraded state | Email, push only |

Priority

Priorities are account-configured (P1–P5 is common). Priority is separate from urgency:

• Urgency — controls notification escalation behavior
• Priority — business classification of impact severity

Alerts vs. Incidents

• Alert — Raw signal from a monitoring integration (many per incident)
• Incident — The grouped, actionable work item (one or more alerts)

list_incident_alerts shows you which monitoring signals triggered an incident.

Field Reference

| Field | Type | Description | |-------|------|-------------| | id | string | Unique incident ID | | incident_number | integer | Human-readable number (#342) | | title | string | Short summary of the incident | | status | string | triggered / acknowledged / resolved | | urgency | string | high / low | | priority | object | Account-configured priority (P1–P5) | | service | object | The service this incident belongs to | | assignments | array | Responders currently assigned | | escalation_policy | object | Escalation policy in effect | | created_at | datetime | When the incident was created | | resolved_at | datetime | When it was resolved (null if open) | | body.details | string | Incident description/details | | alert_counts.triggered | integer | Number of triggered alerts |

Common Workflows

Triage Open Incidents

1. Call list_incidents with statuses[]=triggered&statuses[]=acknowledged, sorted by created_at desc
2. Group by urgency: high urgency first
3. For each critical incident, call get_incident for full details
4. Call list_incident_alerts to understand the triggering signal
5. Call list_past_incidents on the incident ID to find similar historical events
6. Add acknowledgment note via create_incident_note with your investigation steps

Investigate a Triggered Incident

1. Call get_incident to get full details (service, escalation policy, assignments)
2. Call list_incident_alerts to see what monitoring signals fired
3. Call list_past_incidents — this uses AI to find similar past incidents and their resolutions
4. Call list_incident_log_entries for the full audit trail of notifications and escalations
5. Acknowledge via update_incident with status=acknowledged

Bulk Resolve After Maintenance

After a maintenance window resolves multiple incidents:

1. Call list_incidents with statuses[]=triggered&statuses[]=acknowledged filtered to the affected service
2. Call manage_incidents with all incident IDs and status=resolved
3. Add a resolution note to each with create_incident_note

Cross-Vendor PSA Ticket Correlation

PagerDuty incidents often need to be tracked in a PSA for billing/SLA reporting:

1. When a high-urgency PagerDuty incident is created, create a corresponding PSA ticket
2. Store the PagerDuty incident number (#342) in the PSA ticket body for cross-reference
3. Map PagerDuty urgency → PSA priority: high → Critical/High, low → Medium/Low
4. When the PagerDuty incident resolves, update the PSA ticket with the resolution timestamp
5. Use list_incident_log_entries to extract total response time for SLA tracking

Merge Duplicate Incidents

When the same root cause fires multiple incidents:

1. Identify the primary incident (earliest or highest urgency)
2. Call merge_incidents with the primary incident id and the IDs of the duplicates in source_incidents[]
3. PagerDuty merges all alerts and log entries into the primary incident
4. The secondary incidents are automatically resolved

Error Handling

| Error | HTTP Code | Resolution | |-------|-----------|------------| | Invalid token | 401 | Use Token token=<key> format — NOT Bearer | | Incident not found | 404 | Verify ID; use list_incidents to find valid IDs | | Service not found | 404 | Verify service ID with list_services | | Status conflict | 409 | Incident may already be resolved; check current status | | Rate limited | 429 | Back off 60 seconds; PagerDuty limits 900 req/min |

Best Practices

1. Use list_past_incidents — PagerDuty's AI similarity search often surfaces the exact runbook needed
2. Filter by service and team — Always scope incident queries to relevant services to avoid noise
3. Acknowledge before investigating — Stops escalation clock while you triage
4. Add notes as you go — create_incident_note builds a shared investigation timeline
5. Merge duplicates immediately — Reduces responder confusion during active incidents
6. Use since/until for reports — Unbounded queries on large accounts are slow

Related Skills

• [On-Call Management](../oncall/SKILL.md) — Who is on-call, escalation policies, overrides
• [API Patterns](../api-patterns/SKILL.md) — Token format, 66-tool reference, pagination