Experimental Design
Experimental design is the art and science of creating tests that can actually answer the question being asked. A well-designed experiment isolates the variable of interest, controls for confounds, produces data with enough precision and statistical power to discriminate between competing explanations, and does so within ethical and practical constraints. A poorly designed experiment wastes time, money, and trust.
Agent affinity: mcclintock (experiment design), wu (measurement precision)
Concept IDs: sci-controlled-experiments, sci-experimental-controls, sci-variables-types, sci-replication-reliability
The Design Toolbox at a Glance
| # | Design Element | Purpose | Key question | |---|---|---|---| | 1 | Variable identification | Clarify what changes and what is measured | What is the IV? What is the DV? What is held constant? | | 2 | Control groups | Provide a baseline for comparison | What does "no treatment" look like? | | 3 | Randomization | Eliminate systematic assignment bias | Are subjects assigned to conditions by chance? | | 4 | Blinding | Eliminate observer and subject bias | Do participants/observers know which condition they are in? | | 5 | Sample size & power | Ensure the experiment can detect real effects | How many subjects are needed to detect an effect of size X? | | 6 | Replication | Establish reliability | Can the result be reproduced? | | 7 | Factorial design | Test multiple variables and their interactions | Do variables interact, or do they act independently? | | 8 | Quasi-experimental design | Handle situations where true randomization is impossible | How do we make causal claims without random assignment? |
Element 1 -- Variable Identification
Every experiment has three categories of variables:
Independent variable (IV): The factor deliberately manipulated by the experimenter. In a drug trial, this is the drug vs. placebo. In a plant growth study, this might be the wavelength of light.
Dependent variable (DV): The factor measured as the outcome. In the drug trial, this might be blood pressure. In the plant study, this might be stem height after 14 days.
Controlled variables (CVs): All other factors held constant across conditions. In the plant study: soil type, pot size, watering schedule, temperature, photoperiod, seed source.
The single-variable rule: Change one thing at a time. If two variables change simultaneously and the outcome changes, you cannot determine which variable caused the change. This is the most fundamental design principle and the most commonly violated.
Exception: Factorial designs deliberately change multiple variables, but they are designed to detect both individual effects and interaction effects. The single-variable rule applies to simple experiments; factorial designs are a structured relaxation of it.
Element 2 -- Control Groups
A control group receives no treatment (or a baseline treatment) so that the experimental group's outcome can be compared against something.
Negative control: A condition expected to produce no effect. If the negative control shows an effect, something is wrong with the experimental setup (contamination, measurement error, uncontrolled variable).
Positive control: A condition expected to produce a known effect. If the positive control fails to show the expected effect, the experimental system is not working (wrong concentration, dead cells, broken instrument).
Placebo control: In studies involving human subjects, participants in the control group receive an inert treatment (sugar pill, sham procedure) identical in appearance to the real treatment. This controls for the placebo effect -- the tendency for subjects to improve simply because they believe they are being treated.
Why controls matter: Without a control group, an observed outcome is uninterpretable. "The plants grew 15 cm" means nothing. "The plants grew 15 cm compared to 8 cm in the control group under identical conditions except for the experimental variable" means something.
Element 3 -- Randomization
Random assignment of subjects to conditions prevents systematic differences between groups that could confound the results.
Simple randomization: Flip a coin or use a random number generator. Works well for large samples.
Stratified randomization: First divide subjects into subgroups (strata) by a known confound (e.g., sex, age bracket, prior condition), then randomize within each stratum. Ensures balanced representation.
Block randomization: Create blocks of a fixed size (e.g., 4), with each block containing an equal number of treatment and control assignments in random order. Prevents long runs of the same assignment.
Why randomization matters: Without it, subject assignment may correlate with unmeasured variables. If the experimenter (consciously or unconsciously) assigns healthier-looking plants to the treatment group, the treatment will appear to work even if it does nothing.
Element 4 -- Blinding
Single-blind: The subject does not know which condition they are in, but the experimenter does. Prevents subject bias (placebo effect, performance anxiety).
Double-blind: Neither the subject nor the experimenter measuring outcomes knows which condition the subject is in. Prevents both subject and observer bias.
Triple-blind: Subjects, experimenters, and data analysts are all blinded. The data analyst does not know which group is treatment and which is control until after the analysis is complete.
When blinding is impossible: Some interventions cannot be blinded (surgery vs. no surgery, exercise vs. no exercise). In these cases, blinding the outcome assessor (the person measuring the DV) is still valuable.
Element 5 -- Sample Size and Statistical Power
Power is the probability that the experiment will detect a real effect if one exists. Conventionally, power = 0.80 (80% chance of detecting a real effect).
Sample size determines power. Too few subjects = low power = the experiment may miss a real effect (Type II error, false negative). Too many subjects = wasted resources (though never a statistical error).
Power analysis calculates the required sample size given:
- Expected effect size (how large is the difference you want to detect?)
- Significance level (alpha, typically 0.05)
- Desired power (typically 0.80)
- Variability in the measurement (standard deviation)
Rule of thumb: If you do not do a power analysis, you are guessing at sample size. If you are guessing, you are probably guessing low.
Element 6 -- Replication
Within-study replication: Multiple independent measurements under each condition. This is required, not optional. A single measurement per condition is an anecdote.
Between-study replication: Independent researchers repeating the same experiment. This is the gold standard for establishing that a result is real and not an artifact of one laboratory's particular conditions, equipment, or biases.
The replication crisis: Many published scientific findings have failed to replicate (estimates range from 30% to 70% in some fields). Contributing factors: small sample sizes, p-hacking, publication bias toward positive results, insufficient methodological detail. The response: pre-registration (declaring hypotheses and analysis plans before running the experiment), open data, and increased emphasis on replication studies.
Element 7 -- Factorial Design
When two or more independent variables are of interest, a factorial design tests all combinations:
2x2 factorial example: Testing the effect of fertilizer type (A vs. B) and watering frequency (daily vs. weekly) on plant growth.
| | Daily watering | Weekly watering | |---|---|---| | Fertilizer A | Condition 1 | Condition 2 | | Fertilizer B | Condition 3 | Condition 4 |
This design answers three questions simultaneously:
- Does fertilizer type matter? (main effect