scientific-critical-thinking

Scientific Critical Thinking: Evaluating Evidence and Claims

Overview

Scientific critical thinking is the disciplined application of logical and methodological standards to evaluate whether a study's design, analysis, and interpretation support its conclusions. It is the skill that separates a researcher who synthesizes evidence from one who accumulates it. This guide covers the hierarchy of evidence, the mechanics of common biases, effect size interpretation, the p-value controversy, GRADE evidence grading, and common logical fallacies in the interpretation of scientific literature.

Key Concepts

1. Study Design Hierarchy

Study designs vary in their ability to support causal inference. The hierarchy below applies to questions about the effect of an intervention or exposure on an outcome:

Systematic reviews and meta-analyses of RCTs (highest causal certainty)
    ↓
Randomized Controlled Trials (RCTs)
    ↓
Non-randomized controlled trials / cluster-randomized trials
    ↓
Prospective cohort studies (follow exposure → outcome forward in time)
    ↓
Retrospective cohort studies
    ↓
Case-control studies (compare exposed vs. unexposed given outcome)
    ↓
Cross-sectional studies (measure exposure and outcome simultaneously)
    ↓
Case series and case reports
    ↓
Expert opinion, mechanistic reasoning, animal models (lowest causal certainty)

Important exceptions: For questions about rare outcomes, case-control designs are often more efficient than cohort studies. For questions about diagnostic accuracy, randomized designs are usually inappropriate — cross-sectional or cohort designs with verified reference standards are preferred. For harm questions, RCTs are often infeasible (ethical constraints), making large cohort studies the best available evidence.

2. Effect Measures and Their Interpretation

Effect measures quantify the relationship between an exposure/intervention and an outcome. Confusing them is a leading source of misinterpretation.

| Measure | Formula | Use case | Key interpretation | |---------|---------|----------|-------------------| | Risk Ratio (RR) | Risk in exposed / Risk in unexposed | Cohort studies, RCTs | RR = 2.0: exposed group has twice the risk | | Odds Ratio (OR) | Odds in exposed / Odds in unexposed | Case-control studies, logistic regression | Approximates RR when outcome is rare (<10%); overestimates RR for common outcomes | | Hazard Ratio (HR) | Instantaneous event rate ratio | Survival analysis (Cox regression) | HR = 0.7: 30% lower hazard of event per time unit in treated group | | Number Needed to Treat (NNT) | 1 / Absolute Risk Reduction | Clinical decision-making | NNT = 20: treat 20 patients to prevent 1 event | | Absolute Risk Reduction (ARR) | Risk_control − Risk_treated | Clinical impact | ARR = 2%: intervention reduces absolute event rate by 2 percentage points | | Cohen's d | (μ₁ − μ₂) / σ_pooled | Continuous outcomes, psychology | d = 0.2 small; 0.5 medium; 0.8 large | | Pearson r | Correlation coefficient | Association, not causal | r = 0.1 small; 0.3 medium; 0.5 large (Cohen 1988) |

Common error: Reporting only the relative risk reduction (e.g., "50% reduction in risk") without the absolute risk reduction. A treatment that reduces risk from 2% to 1% has a 50% relative reduction but only 1% absolute reduction (NNT = 100). The relative measure appears more impressive but the absolute measure is clinically relevant.

3. Bias Types

Bias is systematic deviation of results or inferences from the truth. Unlike random error (reduced by larger samples), bias is directional and not correctable by increasing sample size.

Selection bias: Systematic difference in characteristics between those selected and not selected for study.

• Examples: Healthy worker effect (workers healthier than general population), loss to follow-up bias (sicker patients drop out), volunteer bias
• Detection: Compare baseline characteristics of included vs. excluded participants; assess attrition patterns

Information bias: Systematic error in measuring exposure or outcome.

• Recall bias: Cases remember exposure better than controls (especially case-control studies for rare diseases)
• Observer bias: Assessors aware of exposure status rate outcomes differently
• Detection: Look for blinding of outcome assessors; validated measurement instruments; objective vs. self-reported outcomes

Confounding: A variable associated with both the exposure and the outcome, creating a spurious or masked association.

• Positive confounding: Confounder inflates the observed association
• Negative confounding: Confounder masks a true association
• Detection: Compare crude and adjusted effect estimates; large change (>10%) indicates confounding
• Control methods: Randomization (RCTs), multivariable adjustment, propensity score methods, restriction, matching

Reporting bias: Selective reporting of outcomes or results based on their statistical significance or direction.

• Publication bias: Positive results are published; null results are not
• Outcome reporting bias: Predefined outcomes not reported if non-significant; unplanned analyses reported if significant
• Detection: Compare registered protocol vs. published outcomes; funnel plot asymmetry for meta-analyses

4. The P-value and Statistical vs. Clinical Significance

A p-value is the probability of observing results at least as extreme as those obtained, under the null hypothesis. It is NOT the probability that the null hypothesis is true, nor the probability that the finding is a false positive.

Correct interpretation: p < 0.05 means that, if the null hypothesis were true, fewer than 5% of equally designed studies would produce results this extreme or more. It does NOT indicate the effect is large, clinically meaningful, or replicable.

Statistical significance ≠ clinical significance: With large enough samples, even trivially small effects become statistically significant. A blood pressure drug that reduces systolic BP by 0.8 mmHg (95% CI 0.3–1.3, p = 0.001) is statistically significant but clinically irrelevant.

Confidence intervals are more informative than p-values: A 95% CI of [0.5 kg, 35 kg weight loss] and a 95% CI of [0.5 kg, 1.5 kg weight loss] can both have p < 0.05, but the clinical implications are vastly different. Always focus on CI width and range, not just whether it excludes the null.

5. GRADE Evidence Certainty

GRADE classifies the certainty of evidence across four levels:

| GRADE level | Meaning | Typical starting point | |-------------|---------|----------------------| | High | Further research very unlikely to change confidence | Consistent RCTs, large effect, no bias | | Moderate | Further research likely to have important impact | RCTs with limitations, or strong consistent observational | | Low | Further research very likely to have important impact | Observational studies, or RCTs with serious limitations | | Very low | Any estimate is very uncertain | Case series, expert opinion, very inconsistent results |

GRADE certainty can be downgraded for: risk of bias, inconsistency (heterogeneity), indirectness (different population/outcome), imprecision (wide CI), and publication bias. It can be upgraded for: large effect (OR > 5), dose-response relationship, or all plausible confounders would reduce the effect.

Decision Framework

How should I evaluate this study?
│
├── Step 1: What question is being answered?
│   ├── Intervention effectiveness → Need RCT or high-quality cohort
│   ├── Diagnostic accuracy → Need cross-sectional vs reference standard
│   ├── Prognosis → Need prospective cohort
│   └── Harm / rare exposure → Case-control or large cohort acceptable
│
├── Step 2: Is the study design appropriate?
│   ├── Design matches question → Proceed
│   └── Mismatch → Major limitation (flag)
│
├── Step 3: What are the key