experiment-metrics

Experiment Metrics Selection: STEDII Framework

When to use: Before launching any experiment, when metrics feel unreliable, or when experiment results are confusing

Framework source: Aakash Gupta's "How to Choose the Right Metrics to Evaluate Experiments"

The STEDII Framework

Choose experiment metrics that are:

1. Sensitive
2. Timely
3. Efficient
4. Debuggable
5. Interpretable
6. Isolated

1. Sensitive (Detects Small But Meaningful Changes)

What it means: The metric moves when your feature actually improves the experience

Bad example:

• Metric: Monthly Active Users (MAU)
• Problem: Too coarse. A good onboarding improvement might not move MAU for months.

Good example:

• Metric: Day 7 activation rate
• Why: Sensitive enough to detect onboarding improvements within a week

How to check: Ask: "If this experiment succeeds, will this metric move within the experiment window?"

Common mistake: Using metrics that are too aggregated (MAU, total revenue) when you need something more granular (daily activation, conversion rate by cohort).

2. Timely (Results Available Quickly)

What it means: You get signal fast enough to make decisions

Bad example:

• Metric: 90-day retention
• Problem: Takes 90 days to know if your experiment worked

Good example:

• Metric: Day 7 retention + leading indicators
• Why: Faster feedback, correlates with long-term retention

Tradeoff alert: Sometimes you NEED slow metrics (LTV, annual retention). In those cases:

• Use leading indicators to get fast signal
• Run smaller experiments to validate
• Accept longer experiment duration for critical decisions

How to check: Ask: "Can I get actionable results within [1 week / 2 weeks / 1 month]?"

3. Efficient (High Statistical Power)

What it means: You can detect the effect with reasonable sample size and time

Bad example:

• Metric: Revenue per user
• Problem: High variance, need massive sample sizes

Good example:

• Metric: Conversion rate
• Why: Lower variance, reaches significance faster

Statistical power explained:

• Power = ability to detect a real effect
• Higher variance metrics = lower power = longer experiments
• Formula: Sample size needed ∝ (Variance / Expected Effect Size)²

How to check: Run a power calculation:

Minimum sample size = (Z + Z)² × (σ² / δ²)
Where:
- Z = confidence level (usually 1.96 for 95%)
- σ = standard deviation of metric
- δ = minimum detectable effect

Practical tip: If you need >1M users to detect a 5% lift, your metric isn't efficient enough.

4. Debuggable (Easy to Diagnose Issues)

What it means: When something goes wrong, you can figure out why

Bad example:

• Metric: "Engagement score" (black box formula)
• Problem: If it drops, you don't know what broke

Good example:

• Metric: Click-through rate (CTR)
• Why: Simple, transparent, easy to debug

How to check: Ask: "If this metric tanks, can I quickly understand what happened?"

What makes metrics debuggable:

• ✅ Simple calculations
• ✅ Can be broken down by segments
• ✅ Can view user-level data
• ✅ Clear numerator and denominator

Red flags:

• ❌ Proprietary "engagement scores"
• ❌ Complex weighted formulas
• ❌ Metrics with 5+ variables
• ❌ Black box ML model outputs

5. Interpretable (Easy to Understand and Explain)

What it means: Stakeholders can understand what the metric represents

Bad example:

• Metric: "Quality-adjusted sessions per visitor"
• Problem: What does "quality-adjusted" mean?

Good example:

• Metric: "% of users who complete onboarding"
• Why: Crystal clear what it measures

The grandma test: Can you explain this metric to your grandma? If not, it fails interpretability.

How to check:

• Can you explain it in one sentence?
• Would a new PM understand it immediately?
• Can executives grasp it without training?

6. Isolated (Measures Only What You Changed)

What it means: The metric moves because of your experiment, not external factors

Bad example:

• Metric: Total signups
• Problem: Could move due to marketing campaigns, seasonality, competitor changes

Good example:

• Metric: Signup conversion rate (for signup flow experiment)
• Why: Isolated to the signup flow you're testing

Common isolation failures:

• Network effects (social features affect all users)
• Cross-contamination (treatment bleeds to control)
• Seasonality (holiday effects)
• Marketing campaigns running simultaneously

How to check: Ask: "Could something OTHER than my experiment cause this metric to move?"

How to Use This Framework

Step 1: List Your Candidate Metrics

Use /experiment-metrics

I'm running an experiment to: [describe your experiment]

Help me brainstorm 5-10 candidate metrics we could measure.

Step 2: Score Each Metric Against STEDII

Create a table:

| Metric | Sensitive? | Timely? | Efficient? | Debuggable? | Interpretable? | Isolated? | Total Score | |--------|------------|---------|------------|-------------|----------------|-----------|-------------| | Metric 1 | 2/3 | 3/3 | 2/3 | 3/3 | 3/3 | 2/3 | 15/18 | | Metric 2 | 3/3 | 1/3 | 3/3 | 2/3 | 3/3 | 3/3 | 15/18 |

Scoring:

• 3 = Excellent
• 2 = Acceptable
• 1 = Poor
• 0 = Fails this criterion

Step 3: Select Primary + Guardrail Metrics

Primary metric: The ONE metric your experiment is designed to move

• Should score 15+/18 on STEDII
• The metric you'll make decisions on

Guardrail metrics (3-5): Metrics you DON'T want to hurt

• Revenue (don't tank it)
• Core engagement (don't break the product)
• Quality metrics (don't hurt user experience)

Example:

• Primary: Day 7 activation rate
• Guardrails: Revenue per user, Daily active users, Customer satisfaction score, Page load time

Step 4: Run Pre-Experiment Checks

Before launching:

1. A:A Test - Run experiment with no actual change
2. Both groups should be identical
3. If metrics differ, you have a setup problem

1. Sample Ratio Check - Verify 50/50 split is actually 50/50
2. If you see 52/48 or worse, investigate

1. Metric Stability - Check historical variance
2. High variance = longer experiment needed

Common Metric Selection Mistakes

Mistake #1: Using Only One Metric

Problem: Optimize one thing, break another

Solution: Always have guardrail metrics

• Primary: what you're trying to improve
• Guardrails: what you don't want to hurt

Mistake #2: Confusing Leading and Lagging Metrics

Lagging metrics:

• Slow to respond
• Ultimate outcome you care about
• Example: LTV, annual retention, NPS

Leading metrics:

• Fast signal
• Predictive of lagging metrics
• Example: Day 7 retention, activation rate

Best practice: Use leading metrics to get fast signal, validate with lagging metrics on a sample.

Mistake #3: Metric Dilution

Problem: Testing a small feature but measuring site-wide metrics

Example:

• Test: New checkout button color
• Metric: Monthly revenue
• Issue: Only 5% of users even see checkout, signal is too diluted

Solution: Measure metrics scoped to exposed users

• Better metric: Revenue per checkout visitor
• Or: Conversion rate (checkout started → completed)

Mistake #4: Simpson's Paradox

Problem: Aggregate metric moves one way, segments move the opposite way

Example:

• Overall conversion rate: +5% ✅
• Mobile conversion: -10% ❌
• Desktop conversion: -5% ❌
• Why? More cheap mobile traffic shifted the mix

Solution: Always segment your metrics (new vs returning, mobile vs desktop, etc.)

Real-World Examples

Example 1: Netflix Thumbnail Test

Experiment: Testing new thumbnail images

Bad metric: Monthly viewing hours

• Not sensitive (too aggregated)
• Not timely (takes too long)
• Not isolated (affected by content releases)

Good metric: Click-through rate on