rca-causal-inference

Causal-Inference RCA

Most root cause analysis is qualitative — you interview people, draw diagrams, write narratives. That's necessary but insufficient for systems where you have rich observational data and need defensible, reproducible, quantitative causal claims.

This skill teaches the mathematical techniques developed by Pearl, Spirtes, Glymour, and others to reason formally about cause from data: when you can identify a causal effect from observations alone, when you need an intervention, and how to compute what would have happened under a counterfactual.

The Ladder of Causation

Judea Pearl (The Book of Why, 2018) defines three levels of causal reasoning, each requiring strictly more machinery than the last:

| Rung | Activity | Example question | Required | |---|---|---|---| | 1 | Association | "How often does latency correlate with CPU?" | Joint distribution P(X, Y) | | 2 | Intervention | "What happens to latency if we force CPU=50%?" | do-calculus, do(X) operator | | 3 | Counterfactual | "Given what happened, would latency have stayed normal if CPU had been 50%?" | Structural causal model + observed data |

Most RCA lives implicitly at rung 3 — every "the root cause was X" statement is a counterfactual — but most RCA tools can only support rung 1. The gap is where spurious root causes come from.

Structural Causal Models (SCMs)

An SCM is a triple (U, V, F) where:

• U — exogenous variables (unobserved background conditions, modeled as independent random variables)
• V — endogenous variables (observed system variables)
• F — a set of structural equations, one per endogenous variable, expressing how each variable is generated from its parents and from exogenous noise

Example for a simple latency incident:

U = {u_traffic, u_cpu_noise, u_lat_noise}
V = {Traffic, CPU, Latency}
F: Traffic := u_traffic
   CPU     := 0.7 * Traffic + u_cpu_noise
   Latency := 0.3 * CPU + 0.1 * Traffic + u_lat_noise

This SCM encodes three claims: Traffic is exogenous, CPU is caused by Traffic plus noise, and Latency is caused by both CPU (primary) and Traffic (direct effect, e.g., queueing). A directed acyclic graph (DAG) corresponds one-to-one:

Traffic → CPU → Latency
       ↓________↑

Why the DAG matters

The DAG encodes which variables can be confounders. Pearl's do-calculus operates on DAGs to answer: "Given the DAG structure and observational data, can I compute the effect of do(X=x) on Y without running an experiment?" The answer depends on whether a valid adjustment set exists — a set of variables that, when conditioned on, blocks all backdoor paths from X to Y.

Backdoor criterion

A set Z satisfies the backdoor criterion for (X, Y) if:

1. No node in Z is a descendant of X.
2. Z blocks every path between X and Y that contains an arrow into X.

If such a Z exists, then:

P(Y | do(X=x)) = Σ_z P(Y | X=x, Z=z) P(Z=z)

This is the adjustment formula. It says: to compute the causal effect of X on Y, stratify by Z and average.

Frontdoor criterion

When no backdoor adjustment set exists (unmeasured confounders), the frontdoor criterion may still rescue you: find a set M such that M is on every directed path X→Y, M is unconfounded with X, and no unblocked backdoor path from M to Y exists. Then:

P(Y | do(X=x)) = Σ_m P(m | x) Σ_{x'} P(Y | m, x') P(x')

Counterfactual computation

Given an SCM and observed evidence e = {X=x, Y=y}, compute "would Y have been y' had X been x'?":

1. Abduction — use evidence e to update the distribution over U.
2. Action — replace the structural equation for X with X := x' (mutilate the graph).
3. Prediction — compute P(Y=y') under the modified SCM and updated U.

This is the only known formal procedure for computing individual-level counterfactuals. Every "the root cause was X" claim you want to defend mathematically reduces to a counterfactual computation in a specified SCM.

Bayesian Networks for fault diagnosis

A Bayesian network is a DAG with a conditional probability table (CPT) at each node. When the CPTs encode causal rather than merely probabilistic dependencies, the network functions as an SCM for computing effects and counterfactuals.

Fault-diagnosis workflow

1. Define the variables: symptoms, components, environmental factors.
2. Draw the DAG from domain knowledge (or learn it — see structure learning below).
3. Estimate CPTs from historical incident data or expert elicitation.
4. At incident time, instantiate observed variables as evidence.
5. Compute posterior P(fault | evidence) using variable elimination, junction tree, or MCMC.
6. Rank candidate faults by posterior probability × remediation cost.

Empirical performance

Paper 2 of our mathematical-foundations research: a comparative evaluation of Bayesian network fault diagnosis across six industrial case studies found classification accuracy of 85–94% when network structure matches the physical process, dropping to 60–70% under structure misspecification. The lesson: the DAG is the hard part. A correct DAG with mediocre CPTs outperforms a wrong DAG with perfect CPTs.

Structure learning

When you don't know the DAG a priori:

• Constraint-based (PC, FCI algorithms by Spirtes, Glymour, Scheines): test conditional-independence claims in the data and rule out edges that contradict them.
• Score-based (GES, BIC-scored search): score candidate DAGs by fit + complexity penalty, hill-climb to a local optimum.
• Hybrid (MMHC): constraint-based skeleton then score-based orientation.

Structure learning from observational data identifies the DAG only up to Markov equivalence — it cannot distinguish X→Y from Y→X without intervention or temporal order. Use it as a hypothesis generator, not as ground truth.

Graph-theoretic fault localization

When you have a service dependency graph (or any causal graph) and want to localize a fault to a node, centrality measures help.

Centrality measures

| Measure | Intuition | Best for | |---|---|---| | Degree centrality | How many neighbors does a node have? | Identifying hubs | | Betweenness | How often does a node lie on shortest paths? | Identifying bottlenecks | | Closeness | How far is a node from all others? | Identifying influential nodes | | Eigenvector / PageRank | How connected are a node's neighbors? | Identifying prestige/importance | | Katz | Weighted walks attenuated by distance | Weighted influence |

Paper 3 finding (our research)

Graph centrality alone is insufficient for fault localization in microservices — it identifies structurally important nodes, but the structurally important node is often not where the fault is. The best-performing methods combine centrality with runtime metrics: weight edges by current request rate and latency, then rank by weighted centrality. This marries the static architectural view with the dynamic operational view.

Information-theoretic methods

When you have high-frequency time series but no reliable DAG, information-theoretic measures can detect statistical coupling without assuming a generative model.

Transfer entropy

TE(X → Y) = H(Y_{t+1} | Y_t^k) - H(Y_{t+1} | Y_t^k, X_t^l)

Transfer entropy measures how much the future of Y is predictable from the past of X beyond what's already predictable from Y's own past. It's directional, nonparametric, and detects nonlinear coupling.

Mutual information

I(X; Y) = Σ_{x,y} P(x,y) log [P(x,y) / (P(x)P(y))]

Symmetric measure of statistical dependence. Not causal by itself — X and Y can have high MI because both depend on a common driver — but useful as a screening tool.

Paper 4 finding

Transfer entropy substantially outperformed lagged correlation for fault localization in a 24-node industrial chemical process. The key advantage: transfer entropy detects nonlinear coupling that linear correlation misses entirely.

Granger causality