statistical-modeling

Statistical Modeling

Statistical modeling is the practice of fitting mathematical structures to data in order to quantify relationships, test hypotheses, and make predictions. Unlike machine learning, which optimizes prediction, statistical modeling privileges interpretability and inference -- understanding why variables relate, not just that they do. Leo Breiman's "two cultures" paper (2001) crystallized this distinction. This skill covers the inferential tradition while acknowledging where the two cultures overlap.

Agent affinity: tukey (EDA and diagnostics), fisher (experimental design and ANOVA), breiman (model comparison)

Concept IDs: data-hypothesis-testing, data-confidence-intervals, data-correlation, data-normal-distribution

The Modeling Workflow

| Stage | Goal | Key operations | |---|---|---| | 1. Formulation | Define the question as a model | Specify response variable, predictors, functional form | | 2. Exploration | Understand data structure | Scatterplots, correlation matrices, distribution checks | | 3. Fitting | Estimate parameters | OLS, MLE, MCMC, IRLS depending on model class | | 4. Diagnostics | Check assumptions | Residual plots, Q-Q plots, leverage, VIF | | 5. Inference | Draw conclusions | Confidence intervals, hypothesis tests, effect sizes | | 6. Selection | Compare models | AIC, BIC, cross-validation, likelihood ratio tests | | 7. Communication | Report results | Effect estimates with uncertainty, not just p-values |

Linear Regression

The Model

y = beta_0 + beta_1 x_1 + beta_2 x_2 + ... + beta_p * x_p + epsilon

where epsilon ~ N(0, sigma^2) independently. The betas are estimated by ordinary least squares (OLS), minimizing the sum of squared residuals.

Assumptions (LINE)

| Assumption | Check | Violation consequence | |---|---|---| | Linearity | Residual vs. fitted plot -- no pattern | Biased estimates, meaningless coefficients | | Independence | Study design, Durbin-Watson test | Underestimated standard errors, inflated significance | | Normality of residuals | Q-Q plot, Shapiro-Wilk test | Unreliable confidence intervals and p-values (less critical for large n by CLT) | | Equal variance (homoscedasticity) | Scale-location plot, Breusch-Pagan test | Inefficient estimates, unreliable standard errors |

Interpretation

• beta_j: The expected change in y for a one-unit increase in x_j, holding all other predictors constant.
• R-squared: Proportion of variance in y explained by the model. Not a measure of model quality alone -- a high R-squared with violated assumptions is meaningless.
• Adjusted R-squared: Penalizes for number of predictors. Always use this for model comparison.

Multicollinearity

When predictors are highly correlated, coefficient estimates become unstable. Variance Inflation Factor (VIF) quantifies this: VIF > 5-10 indicates problematic collinearity. Remedies: drop a predictor, combine predictors via PCA, or use regularization (ridge regression).

Logistic Regression

The Model

For binary outcome y in {0, 1}:

log(p / (1 - p)) = beta_0 + beta_1 x_1 + ... + beta_p x_p

where p = P(y = 1 | x). The left side is the log-odds (logit). Parameters are estimated by maximum likelihood.

Interpretation

• exp(beta_j): The odds ratio for a one-unit increase in x_j, holding other predictors constant. An odds ratio of 1.5 means 50% higher odds of the outcome.
• Predicted probability: p = 1 / (1 + exp(-(beta_0 + beta_1 * x_1 + ...))). The sigmoid function maps the linear predictor to [0, 1].
• No R-squared analog that works well. Use pseudo-R-squared measures (McFadden, Nagelkerke) with caution. Prefer ROC-AUC or calibration plots for assessing fit.

Assumptions

• Observations are independent.
• The log-odds are a linear function of the predictors (check with partial residual plots).
• No perfect multicollinearity.
• No assumption of normality or equal variance -- this is not linear regression with a binary outcome.

Generalized Linear Models (GLMs)

Logistic regression is one instance of the GLM framework. The general structure:

| Component | Role | |---|---| | Random component | Distribution of y (Normal, Binomial, Poisson, Gamma, ...) | | Systematic component | Linear predictor eta = X * beta | | Link function | g(mu) = eta, connecting the mean to the linear predictor |

Common GLMs

| Response type | Distribution | Link | Model name | |---|---|---|---| | Continuous | Normal | Identity | Linear regression | | Binary | Binomial | Logit | Logistic regression | | Count | Poisson | Log | Poisson regression | | Count (overdispersed) | Negative binomial | Log | Negative binomial regression | | Positive continuous | Gamma | Log or inverse | Gamma regression | | Proportion (not 0/1) | Beta | Logit | Beta regression |

Poisson Regression

For count data: log(mu) = beta_0 + beta_1 * x_1 + ... Assumes the mean equals the variance (equidispersion). When variance > mean (overdispersion), use negative binomial or quasi-Poisson. Always check for overdispersion.

Analysis of Variance (ANOVA)

Purpose

ANOVA tests whether group means differ. It is a special case of linear regression where all predictors are categorical.

One-Way ANOVA

• Null hypothesis: mu_1 = mu_2 = ... = mu_k (all group means are equal).
• Test statistic: F = (between-group variance) / (within-group variance).
• Assumptions: Independence, normality within groups, equal variances (Levene's test).
• Post-hoc: If the F-test rejects, pairwise comparisons identify which groups differ. Use Tukey's HSD or Bonferroni correction to control family-wise error rate.

Two-Way ANOVA

Adds a second factor and their interaction. The interaction term tests whether the effect of one factor depends on the level of the other. Always plot the interaction (mean response by factor A, colored by factor B) before interpreting the F-test.

ANOVA as Regression

One-way ANOVA with k groups is equivalent to linear regression with k-1 dummy variables. The F-test in ANOVA is the same as the overall F-test in regression. Understanding this equivalence clarifies that ANOVA is not a separate method -- it is regression with categorical predictors.

Bayesian Methods

The Framework

Bayesian inference updates prior beliefs with data to produce posterior beliefs:

posterior is proportional to likelihood times prior

P(theta | data) proportional to P(data | theta) * P(theta)

Prior Specification

| Prior type | When to use | Example | |---|---|---| | Informative | Strong domain knowledge | "The effect is between 0.5 and 1.5 based on prior studies" | | Weakly informative | Some domain knowledge, want to regularize | Normal(0, 10) for regression coefficients -- centered at zero, wide but not flat | | Non-informative | Want the data to dominate | Uniform or Jeffreys prior. Rarely truly "non-informative" -- all priors carry assumptions |

Posterior Inference

• Credible interval: The 95% credible interval contains the parameter with 95% probability. This is what people think frequentist confidence intervals mean (but they don't).
• Posterior predictive checks: Simulate data from the posterior and compare to observed data. If the model is good, simulated data should look like real data.
• Model comparison: Bayes factors, WAIC, LOO-CV. These naturally penalize complexity without needing a separate penalty term.

Practical Bayesian Workflow

1. Specify the model (likelihood + priors).
2. Fit using MCMC (Stan, PyMC, JAGS) or variational inference.
3. Check convergence: R-hat < 1.01, effective sample size > 400, no divergent transitions.
4. Posterior predictive checks.
5. Report posterior summaries with credible intervals.

Model Selection

Information Criteria

| Criterion | Formula | Interpretation | |---|---|---| | AIC | -2 * log-likelihood + 2k | Estimates out-of-sample prediction error.