Experiment Decision Framework: When to A/B Test vs Ship

Quick Start

/experiment-decision

Then provide:

1. What you're considering building (feature, change, or experiment)
2. Expected impact (metric + estimated improvement)
3. Your concern (is this risky? reversible? controversial?)

I'll walk you through the decision tree: reversibility, hypothesis strength, detectable impact, and risk level. You'll get a clear recommendation: A/B test, ship + monitor, or just ship.

Output: Decision documented inline or saved to thoughts/shared/product/decisions/ Time: ~5 min for clear-cut cases, ~15 min for nuanced decisions

When to use: Before building any feature, when stakeholders demand "data-driven" decisions, or when unsure if testing is worth the effort

Framework source: Aakash Gupta's "When to A/B Test vs Just Ship"

The Decision Framework

Use this decision tree:

Question 1: Is it reversible?

If YES → Ship it

• CSS changes
• Messaging tweaks
• UI polish
• Non-destructive features

Why: Reversible changes have low risk. Ship, monitor, rollback if needed.

If NO → Continue to Question 2

Question 2: Do you have a hypothesis with measurable impact?

If NO → Don't test

• Building "nice to haves"
• No clear success metric
• Can't measure the outcome

Why: Testing without a hypothesis is wasteful. Either clarify the hypothesis or don't build it.

If YES → Continue to Question 3

Question 3: Is the expected impact large enough to detect?

Run a power calculation:

Minimum Detectable Effect (MDE) = Effect you need to see to justify the work

If your feature is expected to improve conversion by 0.5%, but you need 10M users to detect it → Don't test, just ship and monitor

If impact is too small to detect → Ship without test

If impact is detectable → Continue to Question 4

Question 4: Is the risk of being wrong high?

High risk scenarios:

• Affects revenue directly (pricing, checkout)
• Impacts core user experience (onboarding, core flows)
• Controversial decision (stakeholder disagreement)
• Large engineering investment

If HIGH risk → A/B test

If LOW risk → Ship without test

Decision Matrix

When to A/B Test

✅ Test When:

1. High-stakes decisions

• Pricing changes
• Checkout flow modifications
• Core product changes
• Revenue-impacting features

2. Controversial hypotheses

• Team is divided on approach
• Stakeholders disagree
• User research is conflicting

3. Long-term bets

• Features that are expensive to reverse
• Architectural decisions
• Platform changes

4. Optimization work

• Conversion rate improvements
• Engagement optimization
• Retention experiments

When to Just Ship

✅ Ship When:

1. Fast iteration needed

• Competitive pressure
• Time-sensitive opportunities
• Market windows closing

2. Low risk, high certainty

• Bug fixes
• Obvious improvements
• User-requested features (with clear demand)

3. Qualitative insights are strong

• Clear user pain validated through research
• Competitive parity features
• Accessibility improvements

4. Testing would take too long

• Small user base (can't reach significance)
• Slow conversion cycles (months to convert)
• Complex setup (weeks to build test infrastructure)

The Cost of A/B Testing

Time costs:

• Engineering: 2-4 weeks to build test infrastructure
• Analysis: 1-2 weeks to run experiment + analyze
• Total: 3-6 weeks delay

Engineering costs:

• Feature flagging system
• Analytics instrumentation
• A/A test validation
• Test maintenance

Opportunity costs:

• Could have shipped 3-5 other features
• Delayed value delivery to users
• Competitors may ship first

When testing costs exceed value → Just ship

Real-World Examples

Example 1: Amazon's "Add to Cart" Button Color

Decision: A/B Test

• High traffic (millions of users)
• Direct revenue impact
• Easy to detect small improvements
• Result: +2% conversion = $100M+ annually

Example 2: Slack's Message Threading

Decision: Just Ship

• Highly requested feature
• Strong qualitative signal from users
• Reversible (users can ignore threads)
• Result: Successful launch, became core feature

Example 3: Netflix's "Are you still watching?" prompt

Decision: A/B Test

• Controversial (could annoy users)
• Impact on engagement unclear
• Risk of hurting retention
• Result: Test showed improved engagement (prevented zombie sessions)

Common Mistakes

❌ Testing everything "to be data-driven"

• Problem: Slows down velocity
• Fix: Reserve tests for high-stakes decisions

❌ Shipping without monitoring

• Problem: Bad changes go unnoticed
• Fix: Ship with dashboards and alerts

❌ Running underpowered tests

• Problem: Waste time on inconclusive results
• Fix: Calculate sample size before starting

❌ Testing when qualitative data is clear

• Problem: Delays obvious improvements
• Fix: Trust strong user research signals

Quick Reference Checklist

Before building any feature, ask:

• Is this reversible? (If yes → ship)
• Do I have a clear hypothesis? (If no → don't build)
• Can I measure the impact? (If no → don't test)
• Is the expected impact large enough to detect? (Power calculation)
• What's the risk of being wrong? (High risk → test)
• What's the cost of testing vs shipping? (ROI check)
• Do I have strong qualitative data? (If yes → consider shipping)

Statistical Power Guidance

Before committing to an A/B test, estimate whether you have enough traffic to detect a meaningful difference.

Power Calculation Essentials

Three inputs you need:

1. Minimum Detectable Effect (MDE) -- what's the smallest improvement worth detecting?

   • For checkout conversion: 1-2% relative change matters (high revenue impact)
   • For feature adoption: 5-10% relative change is typical MDE
   • For engagement metrics: 3-5% relative change is reasonable

2. Baseline conversion rate -- what's the current rate you're trying to improve?

   • Higher baselines need more samples to detect small changes
   • Lower baselines are easier to move (but may need larger sample)

3. Daily traffic to the experiment -- how many users will enter the test per day?

Rule of Thumb

You need approximately 1,000 conversions per variant to detect a 5% relative change at 80% power (95% confidence).