Machine Learning Foundations
Machine learning is the practice of building systems that learn patterns from data and use those patterns to make predictions or decisions on new data. Where statistical modeling (the inference culture) asks "what is the relationship between X and Y?", machine learning (the prediction culture) asks "given X, what is the best prediction of Y?" This skill covers the foundational concepts, algorithms, and workflow of machine learning from the practitioner's perspective.
Agent affinity: breiman (algorithm selection, ensemble methods), tukey (feature engineering, EDA)
Concept IDs: data-correlation, data-distributions, data-measures-of-spread, data-hypothesis-testing
The ML Workflow
| Stage | Goal | Key operations |
|---|---|---|
| 1. Problem framing | Define the task precisely | Classification vs. regression vs. clustering; define target variable and success metric |
| 2. Data collection | Assemble training data | Sources, sampling, labeling; ensure data represents the deployment population |
| 3. Feature engineering | Create informative inputs | Domain-driven features, transformations, encoding categoricals |
| 4. Train/test split | Prevent overfitting evaluation | Hold out 20-30% for testing; never touch test set during development |
| 5. Model selection | Choose algorithm family | Based on data size, interpretability needs, problem structure |
| 6. Training | Fit model parameters | Optimization (gradient descent, tree splitting, etc.) |
| 7. Validation | Tune hyperparameters | k-fold cross-validation on training set only |
| 8. Evaluation | Assess on held-out test set | Metrics appropriate to the problem (accuracy, F1, RMSE, etc.) |
| 9. Interpretation | Understand what the model learned | Feature importance, partial dependence, SHAP values |
| 10. Deployment | Put the model in production | Monitoring, drift detection, retraining schedule |
Supervised Learning
Classification
The task: given features X, predict a categorical label y.
Key algorithms:
| Algorithm | Strengths | Weaknesses | When to use |
|---|---|---|---|
| Logistic regression | Interpretable, fast, probabilistic | Linear decision boundary | Baseline; when interpretability matters |
| k-Nearest Neighbors | Non-parametric, no training phase | Slow at prediction, curse of dimensionality | Small datasets, low dimensionality |
| Decision tree | Interpretable, handles mixed types | Overfits easily, unstable | When interpretability is paramount; as building block for ensembles |
| Random forest | Robust, handles high dimensions | Less interpretable than single tree | Default for tabular data |
| Gradient boosting | State-of-the-art tabular performance | Prone to overfitting without tuning | Competition-grade tabular prediction |
| SVM | Effective in high dimensions | Slow on large datasets, kernel choice | Text classification, small-medium datasets |
| Neural network | Learns complex patterns, scales to huge data | Requires large data, expensive, black box | Images, text, sequences, very large datasets |
Regression
The task: given features X, predict a continuous value y.
Same algorithms apply (linear regression, k-NN regression, decision tree regression, random forest regression, gradient boosting regression, neural network regression). The loss function changes from cross-entropy to squared error (or absolute error, Huber loss, etc.).
Evaluation Metrics
Classification:
| Metric | Formula / Definition | When to use |
|---|---|---|
| Accuracy | Correct / Total | Balanced classes only |
| Precision | TP / (TP + FP) | Cost of false positives is high (spam detection) |
| Recall | TP / (TP + FN) | Cost of false negatives is high (cancer screening) |
| F1 score | 2 * Precision * Recall / (Precision + Recall) | Need balance between precision and recall |
| ROC-AUC | Area under ROC curve | Ranking quality across thresholds |
| Log loss | Negative log-likelihood of predicted probabilities | When calibrated probabilities matter |
Regression:
| Metric | Formula / Definition | When to use |
|---|---|---|
| MSE | Mean of (y - y_hat)^2 | Default; penalizes large errors |
| RMSE | sqrt(MSE) | Same scale as y; more interpretable |
| MAE | Mean of | y - y_hat |
| R-squared | 1 - (SS_res / SS_tot) | Proportion of variance explained |
| MAPE | Mean of | y - y_hat |
The Bias-Variance Tradeoff
The expected prediction error decomposes into three components:
Error = Bias^2 + Variance + Irreducible noise
- Bias: Error from oversimplifying the model. A linear model fit to a quadratic relationship has high bias (underfitting).
- Variance: Error from model sensitivity to training data. A deep decision tree memorizes the training set and varies wildly across samples (overfitting).
- Irreducible noise: Inherent randomness in the data. No model can reduce this.
The tradeoff: Increasing model complexity reduces bias but increases variance. Decreasing complexity reduces variance but increases bias. The optimal model balances both.
Regularization controls this tradeoff by penalizing complexity:
| Method | Penalty | Effect |
|---|---|---|
| Ridge (L2) | Sum of beta_j^2 | Shrinks coefficients toward zero; keeps all predictors |
| Lasso (L1) | Sum of | beta_j |
| Elastic net | Alpha * L1 + (1 - Alpha) * L2 | Combines ridge and lasso benefits |
| Tree depth limit | Max depth, min samples per leaf | Prevents tree from memorizing noise |
| Dropout | Randomly zero out neurons during training | Prevents neural network co-adaptation |
| Early stopping | Stop training when validation error increases | Universal; works for any iterative algorithm |
Cross-Validation
Cross-validation estimates out-of-sample performance using only the training data.
k-Fold Cross-Validation
- Split training data into k equal folds (k = 5 or 10 is standard).
- For each fold i: train on all folds except i, evaluate on fold i.
- Average performance across all k evaluations.
- Use this average to select hyperparameters.
Critical Rules
- Never use test data for any decision during training. The test set is opened exactly once, at the very end.
- Stratified k-fold for classification. Preserve class proportions in each fold.
- Group k-fold for grouped data. If observations belong to groups (e.g., multiple images from the same patient), all observations from a group must be in the same fold.
- Time series split for temporal data. Training set always precedes validation set in time. No random shuffling.
Decision Trees
How They Work
A decision tree recursively partitions the feature space by choosing splits that maximize information gain (classification) or minimize mean squared error (regression).
Splitting criteria for classification:
- Gini impurity: G = 1 - sum(p_k^2). Measures probability of misclassification.
- Entropy: H = -sum(p_k * log(p_k)). Information-theoretic measure of impurity.
- In practice: Gini and entropy produce nearly identical trees. Gini is slightly faster to compute.
Controlling complexity:
| Parameter | Effect |
|---|---|
| Max depth | Limits tree height; primary regularization lever |
| Min samples split | Minimum observations to attempt a split |
| Min samples leaf | Minimum observations in a terminal node |
| Max features | Number of features considered at each split (critical for random forests) |
Why Single Trees Overfit
A fully grown tree achieves 100% training accuracy by creating one leaf per observation. This is pure memorization. The tree's variance is enormous -- small changes in training data produce completely different trees. This instability is why ensemble methods exist.