Experiment Design
A senior product manager's playbook for running experiments that produce trustworthy decisions.
The default state of experimentation in most companies is sloppy. PMs run tests against vague hypotheses, look at results too early, ignore guardrails, stratify into noise, and ship features whose lift is mostly measurement error. The cost is real: ship the wrong thing, kill the right thing, learn the wrong lesson, repeat.
This skill is the discipline that prevents most of those mistakes. It assumes you have a working experimentation platform (Statsig, PostHog, GrowthBook, Optimizely, Amplitude, Eppo, Kameleoon; the platform does not matter for the principles). It assumes you have product-design and engineering pipelines that can deliver real treatment changes. The hard part is the thinking, and that is what is here.
When to use this skill: any time you are about to design or interpret an experiment. Read the relevant section before you start, not after the test is running.
What this skill covers
The skill spans the full experiment lifecycle. Pre-experiment readiness (is this thing even worth testing). Hypothesis design (cause, effect, magnitude, mechanism). Sample size and minimum detectable effect (do you have enough traffic to learn anything). Duration (how long is long enough, when does the cycle bias the result). Running discipline (no peeking, guardrails, sequential testing). Interpretation (the three buckets and the inconclusive case). Decision-making (matching the result to a pre-committed rule).
The skill does not cover feature flag operational mechanics; those live in the feature-flagging skill, which handles flag taxonomy, environment management, and stale-flag cleanup as a separate discipline. The skill does not cover statistical analysis depth; for delta methods, variance reduction techniques like CUPED, and Bayesian alternatives, see the experimentation-analytics skill. The skill does not cover platform-specific tooling; for MCP commands, auth models, and platform-specific configuration, consult the chosen platform's official documentation. This skill produces the experiment design; the platform implements it.
For the orchestration layer above (which experiments to run, in what order, with what cadence), see the forthcoming experimentation-platform-orchestrator skill. That skill schedules; this skill designs.
The framework: 12 considerations for trustworthy experiment results
A defensible experiment design sits at the intersection of twelve considerations. Each is covered in detail in its own section below.
- Hypothesis discipline. Cause, effect, magnitude, and mechanism. The hypothesis names what is being tested, what should move, by how much, and why.
- Sample size and minimum detectable effect (MDE). Whether the test has enough traffic to detect the effect at the chosen power. Refuse to run underpowered tests.
- Test duration. Longer of the sample-size-hit duration and a full weekly cycle. UI/UX changes need at least 14 days regardless.
- What NOT to A/B test. UX bugs, legal-required changes, brand-philosophy questions, decisions already made, designs whose randomization cannot be clean.
- Segment analysis. Pre-registered segments are evidence; post-hoc segments are noise mining. The multiple comparisons problem is real.
- Interaction effects. Concurrent tests on the same surface can interfere. Mutex enforcement or coordination required.
- Ratio metrics and variance estimation. Naive variance estimators on ratios understate uncertainty. Confirm the platform uses a ratio-aware estimator.
- Network effects and two-sided markets. Treatment can leak into control via interference. Cluster randomization, switchback, or geographic isolation when needed.
- Sequential testing and the peeking problem. Daily peeking inflates false positive rates. Use sequential testing methods when available; pre-commit otherwise.
- Pre-commitment vs p-hacking. Write down the primary metric, MDE, duration, segments, and decision rule before launch. Apply mechanically when results come in.
- Reading results and making the call. Three buckets: clear win, clear loss, inconclusive. The inconclusive bucket exists for a reason; resist the pull to ship anyway.
- Common failures and fixes. A short rapid-fire pattern catalog, expanded in
references/common-failures.md.
The sections below cover each consideration in turn. Read the relevant section before running the experiment, not after.
Hypothesis discipline
The most important section in the skill. Most experiment failures trace back to a vague hypothesis.
A real hypothesis has four parts: cause, effect, magnitude, mechanism. Cause is the change you are making. Effect is the metric you expect to move. Magnitude is how much you expect it to move and from what baseline. Mechanism is why you expect this change to produce this effect.
Bad hypothesis, common shape: "We think the new pricing page will increase conversions." What is wrong with it: no magnitude (how much), no mechanism (why), and the metric is "conversions" rather than a specific event with a clear definition. The team will run this test, look at the result, and argue about what counts as a win. Pre-commitment is impossible because nothing was committed.
Good hypothesis, same domain: "Replacing the three-tier pricing comparison with a single recommended tier will increase signup-to-paid conversion by 8 percent (currently 12 percent, target 13 percent) by reducing decision friction for users who already know they want to subscribe." Cause is the tier replacement. Effect is signup-to-paid conversion, defined as the user reaches the paywall and completes payment within seven days. Magnitude is 8 percent relative lift, taking the rate from 12 to 13 percent absolute. Mechanism is decision friction reduction. Now the team has something to test, a number to hit, and a story to falsify.
Primary metric vs guardrails. The primary metric is the thing you are trying to move. Guardrails are the things that must not break: revenue, retention, support ticket volume, page load time, error rates. Pick exactly one primary metric. Pick three to five guardrails. Multiple primary metrics destroy the discipline because they let you cherry-pick the favorable one when results come in.
Falsifiability test. Before launching the experiment, write down what would make you NOT ship this. If the answer is "nothing, we are committed to the change regardless," the hypothesis is not real and the experiment is theater. Skip the test, save the engineering time, and just ship the change.
Directional vs magnitude distinction. Knowing the change moves the needle is different from knowing it moves the needle enough to matter. A 0.3 percent absolute lift on signup conversion may be statistically significant with enough traffic and still not justify the engineering cost of maintaining the change. Magnitude matters as much as direction; the hypothesis names the magnitude that would justify shipping.
For templates and worked examples across common metric types, see references/hypothesis-templates.md.
Sample size and minimum detectable effect
Sample size grows with the inverse square of the effect you want to detect. Detecting a 1 percent lift requires roughly one hundred times the sample needed to detect a 10 percent lift. Most PMs underestimate this.
The basic decision rule: if your minimum detectable effect (MDE) at current traffic and a reasonable test duration is greater than 5 percent absolute lift, you probably need a bigger MDE. Tiny changes that need huge samples to detect are usually not worth shipping anyway. The change is small either because the underlying mechanism is weak or because the implementation is timid. A weak mechanism is not worth a launch. A timid implementation shou