rca-distributed-systems

Distributed-Systems RCA

Distributed systems fail in ways that classical RCA techniques were not designed to handle. A typical production incident involves 10–100 services, asynchronous messaging, multi-tenant resource contention, tail-latency amplification, and retries that turn a small blip into a cascade. "The root cause was a slow query" is almost always incomplete — you need to explain why this particular slow query caused a user-visible outage when other slow queries didn't.

This skill teaches the techniques that actually work on modern distributed production systems, drawing from the survey literature (Soldani et al. 2023, DynaCausal 2024) and the Microsoft AgentRx work on AI-agent failure diagnosis.

What makes distributed-systems RCA hard

| Property | Consequence for RCA | |---|---| | Scale | Hundreds of services, millions of requests — cannot inspect each one | | Partial visibility | Logs, metrics, traces each capture different slices; none is complete | | Tail amplification | Small perturbations become large outcomes via queueing and retries | | Asynchronous coupling | Cause and effect are not temporally adjacent | | Dependency opacity | Service owners often don't know their downstream consumers | | Blast radius varies | Same fault can affect 0.1% or 100% of users depending on state | | Multi-tenant noise | Other workloads distort the signal | | Feedback loops | Circuit breakers, autoscaling, and retries create nonlinear dynamics |

Classical techniques assume you can identify a failing component. In microservices, everything is slightly degraded and the question is which degradation caused the user-visible symptom.

The three-pillar approach: logs, metrics, traces

Modern observability tools (OpenTelemetry-based stacks) instrument three telemetry types:

• Metrics — low-cardinality numeric time series (CPU, latency percentiles, error rate). Cheap to store, fast to query.
• Logs — high-cardinality event records with structured context. Expensive but precise.
• Traces — causally linked spans representing one request's journey across services. Moderate cost, irreplaceable for causality.

Correlation key

The three pillars are only useful if they share a correlation key. OpenTelemetry's convention:

trace_id = 16-byte ID — one per request, propagated across services
span_id  = 8-byte ID — one per unit of work within a trace
baggage  = key-value metadata that flows with the request

Logs and metrics must carry trace_id in attributes/labels. Without this, you have three independent datasets, not a unified observability signal.

The correlation workflow

1. User reports "checkout is slow."
2. Find an affected request — extract its trace_id from the frontend log.
3. Open the trace in a trace viewer — see the waterfall across services.
4. Identify the slow span — note its service, endpoint, and time window.
5. Pivot to metrics for that service — is this an outlier or broad?
6. Pivot to logs for that service filtered by trace_id — see context.
7. Pivot again: logs for the service during the time window (not just trace_id)
   — look for patterns across many affected requests.
8. Build a hypothesis. Verify against more traces.

The art is knowing when to pivot. Stuck in one pillar for too long is a common failure mode.

Trace-based causality

Traces are the only telemetry type that preserves causal structure. A metric tells you latency went up; a trace tells you which downstream call the latency came from.

Span attributes for RCA

A well-instrumented span includes:

• service.name, service.version
• http.method, http.route, http.status_code
• db.system, db.statement (sanitized)
• rpc.system, rpc.service, rpc.method
• messaging.system, messaging.destination
• exception.type, exception.message, exception.stacktrace
• Custom business attributes (user tier, tenant ID, feature flag values)

Missing or low-cardinality attributes cripple RCA. When building instrumentation, optimize for the attributes you'll need during incident investigation, not for what's easy to emit.

Sampling

You can't trace every request in a high-throughput system. Two common approaches:

• Head-based sampling — decision at span creation. Simple, but misses rare errors.
• Tail-based sampling — decision at span completion, based on full trace content. Keeps all errors and slow traces. More expensive.

For RCA, tail-based sampling is essentially mandatory — head-based sampling discards the exact traces you need.

Service dependency graph analysis

The service graph is the microservice equivalent of a component diagram. Modern service meshes (Istio, Linkerd) and APM tools (Datadog, Honeycomb, New Relic) build it automatically from trace data.

Graph-based RCA techniques

Node-level: for each service, compute centrality measures weighted by current request rate.

Edge-level: for each service-to-service edge, compute error rate and latency. Rank by edge_error_rate × edge_traffic_share × downstream_impact.

Graph diff: compare the dependency graph during the incident to a recent healthy window. New edges, missing edges, or capacity ratio changes often localize the fault quickly.

The common-cause trap

A slow service A might appear as "the cause" in every trace because it sits on every critical path, but the actual fault is an even deeper service B that only A calls. Look for the highest-fanout failing edge, not the most-visible failing node.

DynaCausal — dynamic causality-aware RCA

DynaCausal (Zhang et al., 2024) introduces the idea that causal structure in microservices is itself dynamic — the same service graph can have different causal relationships under different traffic patterns, cache states, and feature-flag configurations. Traditional static graph methods miss this.

Key insight

Learn a time-varying causal graph from trace data, not a single static graph. Use the learned graph at incident time (not a month-old cached graph) to localize the fault.

Implementation sketch

1. Collect trace data in sliding windows (e.g., 5-minute windows).
2. For each window, learn a DAG over service-level metrics using
   constraint-based structure learning (PC algorithm or similar)
   seeded with the known service topology.
3. At incident time, compare the current-window DAG to the historical
   distribution of DAGs. Structural changes (new edges, edge weight
   shifts) point to the causal location.
4. Rank candidate root-cause nodes by their position in the DAG and
   their anomaly severity.

DynaCausal outperformed static baselines by 15–25 percentage points on F1 in the 2024 SockShop and TrainTicket benchmarks.

AgentRx — AI agent failure diagnosis

Microsoft Research's AgentRx (2024) treats AI agents (LLM-driven tool users, workflow agents) as a new class of system where failures arise from reasoning errors, tool-use errors, and planning failures rather than infrastructure faults. The team identified a taxonomy of failure types through trajectory analysis.

The AgentRx failure taxonomy

| Category | Description | Diagnostic signal | |---|---|---| | Reasoning failure | Model produced incorrect logical step | Reasoning chain contradiction | | Planning failure | Wrong decomposition of task into subtasks | Missing required subtask | | Tool misuse | Called a tool with wrong args or for wrong purpose | Tool call outside usage spec | | Grounding failure | Hallucinated entities or values not in context | Fact-check mismatch | | Loop / stall | Agent repeats the same action indefinitely | Self-similarity in trajectory | | Premature termination | Agent stops before task is complete | Missing terminal state | | Context overflow | Context window filled with noise | Token budget exhaustion | | Prompt-injection compromise | Agent followed an adversarial instruction | Authority-check mis