Rootly Incidents

Rootly Incident Management

Overview

Incidents are the primary resource in Rootly — the central record for any production issue, outage, or service degradation. Rootly is used by SRE and platform engineering teams to coordinate response in real time and drive continuous improvement via postmortems. In MSP environments, Rootly typically manages internal infrastructure incidents rather than per-client ticketing, but can be configured with team-based routing to support multi-customer workflows.

The incident system supports:

• Automated Detection - Incidents created from monitoring alert integrations (Datadog, PagerDuty, Prometheus, Grafana, Sentry, etc.)
• Manual Creation - On-call engineers and responders create incidents via Slack, web UI, or API
• AI-Assisted Analysis - find_related_incidents and suggest_solutions surface past patterns and recommendations
• Real-Time Coordination - Slack channel auto-creation, Zoom bridges, timeline tracking
• Post-Incident Learning - Automated postmortem generation and action item tracking

All read and write operations on incidents are available through the Rootly MCP tools.

MCP Tools

Core Incident Tools

| Tool | Description | Key Parameters | |------|-------------|----------------| | incidents_get | List and search incidents | status, severity, page[number], page[size] | | incidents_post | Create a new incident | title, severity_id, team_ids, service_ids | | incidents_by_incident_id_alerts_post | Attach an alert to an incident | incident_id, alert data | | incidents_by_incident_id_alerts_get | List alerts attached to an incident | incident_id | | incidents_by_incident_id_action_items_post | Create an action item on an incident | incident_id, summary, assignee_id | | incidents_by_incident_id_action_items_get | List action items on an incident | incident_id |

AI Analysis Tools

| Tool | Description | Key Parameters | |------|-------------|----------------| | find_related_incidents | Find historically similar incidents using TF-IDF | query or incident_id | | suggest_solutions | Suggest remediation steps based on past resolutions | incident_id or description |

Supporting Lookups

| Tool | Description | |------|-------------| | severities_get | List configured severity levels (ID, slug, color) | | services_get | List services to scope incident to the right owner | | teams_get | List teams for incident assignment and routing | | incident_types_get | List incident types (bug, outage, performance, etc.) | | environments_get | List environments (production, staging, etc.) | | users_get | List users for assignment |

Discover Available Tools

Call list_endpoints to get the current list of all available API endpoints and tool names from the Rootly OpenAPI specification.

Key Concepts

Incident Lifecycle

┌───────────┐   Triage starts   ┌────────────┐   Contained   ┌───────────┐
│ detected  │ ─────────────────> │  in_triage │ ────────────> │ mitigated │
└───────────┘                   └────────────┘               └───────────┘
                                                                     │
                                                               Full fix done
                                                                     ▼
                                                               ┌──────────┐
                                                               │ resolved │
                                                               └──────────┘
                                                                     │
                                                             PIR complete
                                                                     ▼
                                                               ┌────────┐
                                                               │ closed │
                                                               └────────┘

Key lifecycle timestamps on the incident record:

| Timestamp | Meaning | |-----------|---------| | detected_at | Alert fired or issue first observed | | acknowledged_at | Responder acknowledged the page | | in_triage_at | Active investigation started | | started_at | Response team coordinating | | mitigated_at | Immediate impact contained | | resolved_at | Issue fully resolved | | closed_at | Post-incident review complete | | cancelled_at | False alarm; incident cancelled |

Severity Levels

Severities are configurable per Rootly organization. Common conventions:

| Severity | Typical Name | Description | SLA Target | |----------|-------------|-------------|------------| | SEV-1 / Critical | P1 | Complete outage or data loss; business-critical impact | Immediate (15 min) | | SEV-2 / High | P2 | Major feature degraded; significant user impact | 30 minutes | | SEV-3 / Medium | P3 | Partial degradation; workaround available | 2 hours | | SEV-4 / Low | P4 | Minor issue; minimal user impact | Next business day |

Note: Severity IDs are UUIDs in Rootly. Always call severities_get to map severity slugs to IDs before creating an incident.

Incident Status Values

Rootly uses free-text status descriptors alongside timestamps. Common values:

• detected — Newly created, not yet acknowledged
• in_triage — Actively being investigated
• mitigated — Impact contained, monitoring for recurrence
• resolved — Issue fixed, normal operation restored
• cancelled — False alarm or invalid incident

Field Reference

| Field | Type | Description | |-------|------|-------------| | id | string | UUID of the incident | | sequential_id | integer | Human-readable incident number (e.g., INC-342) | | title | string | Short summary of the incident | | summary | string | Detailed description of impact and current status | | status | string | Current lifecycle status | | severity | object | Severity record with slug, color, description | | services | array | Affected services | | environments | array | Affected environments (production, staging) | | teams | array | Teams assigned to respond | | labels | array | Custom tags for filtering and reporting | | started_at | datetime | When the incident started | | resolved_at | datetime | When the incident was resolved (null if open) | | slack_channel_id | string | Auto-created Slack channel for coordination | | url | string | Web UI link to the incident |

Common Workflows

Triage New Incidents

1. Call incidents_get with status=in_triage or status=detected, sorted by severity
2. For each high-severity incident, review title and summary
3. Call find_related_incidents with the incident_id to surface similar past incidents
4. Call suggest_solutions with the incident_id to get AI-generated remediation suggestions
5. Create action items via incidents_by_incident_id_action_items_post for each remediation step
6. Update status as the incident progresses

Create an Incident

1. Call severities_get to find the correct severity ID for the impact level
2. Call services_get to identify which service is affected
3. Call teams_get to find the on-call team to assign
4. Call incidents_post with title, severity_id, service_ids, team_ids
5. Attach any triggering alerts via incidents_by_incident_id_alerts_post

Investigate a Specific Incident

1. Find the incident via incidents_get or by sequential_id
2. Call find_related_incidents — look for recurring patterns (same service, same time of day, same symptoms)
3. Call suggest_solutions — surface past resolutions that worked for similar incidents
4. Review action items via incidents_by_incident_id_action_items_get
5. Review attached alerts via incidents_by_incident_id_alerts_get

Daily Incident Review

1. Call incidents_get with a 24-hour window and status=resolved
2. Count by severity for a daily health summary