Stanford CS329H: Machine Learning from Human Preferences | Autumn 2024 | Ethics

This lecture teaches a powerful technique called regularization, which helps machine learning models avoid overfitting. Overfitting happens when a model learns not only the true patterns in the training data but also the random noise. As a result, it performs well on the training set but poorly on new, unseen data. Regularization fixes this by gently punishing complex models, nudging them to be simpler and more stable. The core idea is simple: add a penalty to the cost (loss) function that grows when the model’s parameters (weights) are large.

The lecture focuses on two main types of regularization: L2 regularization (also called Ridge) and L1 regularization (also called Lasso). With L2, you add a term equal to the sum of the squares of the weights, scaled by a hyperparameter called lambda (λ). This makes all weights smaller but rarely exactly zero. With L1, you add a term equal to the sum of the absolute values of the weights, again scaled by lambda. This version can push some weights exactly to zero, which means the model completely ignores those features. Because of this, L1 naturally performs feature selection. There’s also a combined approach called Elastic Net, which mixes L1 and L2 and requires tuning two hyperparameters.

The lecture uses Mean Squared Error (MSE) as the base cost function example. MSE measures how far predictions are from true values, by averaging the squared differences. Regularization adds a penalty term to MSE, so the optimizer tries to minimize both the prediction errors and the size of the weights. This trade-off is controlled by lambda: a bigger lambda means more penalty on large weights and a simpler model, while a smaller lambda means less penalty and a more flexible model. Lambda is a hyperparameter, which means you choose it rather than the model learning it during training.

To build strong intuition, the instructor uses a geometric picture with two weights (W1 and W2) so we can draw everything in two dimensions. The loss function looks like a set of nested ellipses centered at the best-fit point. L2 regularization acts like putting a circle around the origin; you must pick the best point inside that circle. The optimal point ends up at the place where a loss ellipse just touches the circle—like a tire kissing a curb. Making the circle smaller (increasing lambda) forces the solution closer to the origin, shrinking the weights. With L1, the constraint is a diamond; the optimal point often lands on a corner of the diamond, which lies on the axes, meaning one weight becomes exactly zero. This explains why L1 encourages sparse solutions.

This lecture is for beginners and intermediates who want a practical and intuitive understanding of why and how regularization helps generalization. You should know what a cost function is, what model parameters (weights) are, and have a basic grasp of MSE and the bias-variance tradeoff. If you remember that overfitting means too much variance and not enough bias, regularization will make sense because it increases bias slightly to reduce variance a lot.

After this lecture, you will be able to explain what regularization is, name and describe L1 and L2 (Lasso and Ridge), and understand why L1 produces feature selection while L2 produces smooth shrinkage. You will be able to describe the role of lambda, why it is a hyperparameter, and what happens when you increase or decrease it. You will also be able to visualize regularization using the circle (L2) and diamond (L1) constraints and the tangency with loss ellipses. While the lecture centers on linear models and MSE, the same ideas extend to many other models and loss functions.

The structure of the lecture begins by recalling overfitting and the bias-variance tradeoff. It then introduces the basic idea of regularization as a penalty added to the loss. Next, it defines L2 and L1 regularization precisely, explains lambda as a hyperparameter, and highlights the practical consequences: prediction quality versus feature selection. Finally, it cements understanding with a clear geometric visualization: ellipses for loss, a circle for L2, a diamond for L1, and the solution at the point of tangency inside the constraint. The closing note reminds you that other tools like cross-validation help pick lambda well, and that regularization is a central technique for building models that generalize.

Overall Architecture/Structure

• Data and Model: You start with input features X and target values y. A model, such as linear regression, makes predictions y_hat = X·w (plus possibly a bias term). The training goal is to choose weights w that make y_hat close to y.
• Base Cost Function: The base cost function often used in regression is Mean Squared Error (MSE). In a simple form, J_data(w) = (1/(2N)) Σ_i (y_hat_i − y_i)^2, where N is the number of data points. This measures average squared error between predicted and true values.
• Adding Regularization: Regularization adds a penalty term that grows with the size of w. For L2, the penalty is (λ/(2N)) Σ_j w_j^2. For L1, the penalty is (λ/N) Σ_j |w_j|. The full objective becomes J_total(w) = J_data(w) + J_penalty(w).
• Optimization Objective: The training algorithm no longer minimizes just data error, but the sum of data error and penalty. This balances fitting the data with keeping weights small (simple model). The balance strength is controlled by the hyperparameter lambda (λ).
• Geometric Constraint View: Minimizing J_total(w) is equivalent to minimizing J_data(w) subject to a constraint on w. For L2: Σ_j w_j^2 ≤ C (a circle in 2D, a sphere in higher dimensions). For L1: Σ_j |w_j| ≤ C (a diamond in 2D, a cross-polytope in higher dimensions). The solution is at the tangency of the lowest data-loss contour and the constraint set.

Data Flow

1. Input features X and targets y feed into the model to compute predictions y_hat. 2) Compute data loss: MSE across all data points. 3) Compute penalty: L2 or L1 based on current weights and chosen lambda. 4) Sum them to get total loss J_total. 5) The optimizer updates weights to reduce J_total. 6) Iterate until convergence or stopping criteria are met.

Code/Implementation Details (Conceptual, works across libraries)

• Language/Framework: Any ML framework (scikit-learn, PyTorch, TensorFlow) supports L1/L2 penalties. The math is the same: add a penalty to the loss.
• L2 (Ridge) in Practice: In linear regression, ridge regression solves (X^T X + λI) w = X^T y, where I is the identity matrix. This closed-form solution shows how L2 stabilizes inversion by adding λ to the diagonal, improving numerical conditioning. In gradient descent, you update w ← w − η(∇J_data + (λ/N)w), where η is the learning rate.
• L1 (Lasso) in Practice: L1 lacks a simple closed-form due to the absolute value, which creates kinks at zero. Optimizers use methods like coordinate descent, proximal gradient, or subgradient techniques. The key behavior is that optimality conditions favor exact zeros for some coordinates.
• Elastic Net: Uses a mix of L1 and L2 penalties. Many libraries implement parameters alpha (overall strength) and l1_ratio (how much of alpha goes to L1). For example: total penalty = alpha * (l1_ratio * L1 + (1 − l1_ratio) * L2).
• Parameters and Hyperparameters: Weights w are learned by minimizing J_total. Lambda (and Elastic Net’s split) are hyperparameters you choose—often by trying multiple values and keeping the best one according to a validation metric.

Important Parameters and Meanings

• Lambda (λ): Strength of the penalty. Larger λ means stronger shrinkage (and, for L1, more zeros). Smaller λ means weaker regularization.
• N (number of samples): Standardizing by N keeps penalty comparable as dataset size changes. The formulas in the lecture divide by N or 2N, but the minimizer is unaffected by constant scalings.
• D (number of features): The penalty sums over all D weights. High-D problems benefit strongly from regularization to avoid overfitting.

Optimization Flow

1. Initialize weights w (e.g., zeros or small random values). 2) At each iteration, compute predictions y_hat. 3) Compute data loss and penalty. 4) Compute gradients (or subgradients for L1). 5) Update weights using an optimizer (gradient descent, coordinate descent, etc.). 6) Repeat until loss stabilizes or a stopping criterion is met.

Tools/Libraries Used (Common Options)

• Scikit-learn (Python): ridge regression (Ridge), lasso regression (Lasso), and ElasticNet classes for regression. These accept alpha (similar to λ) and, for ElasticNet, l1_ratio. Fit with .fit(X, y) and predict with .predict(X).
• PyTorch/TensorFlow: Define the base loss (e.g., MSE) and add weight decay (L2) by including λ Σ w^2 either in the loss or as an optimizer parameter (weight_decay). L1 can be added manually by summing |w| across parameters and adding to loss.
• General Tip: Standardize or normalize features before applying regularization so that penalties treat features fairly (this is especially important for L1).

Step-by-Step Implementation Guide

• Step 1: Prepare data. Split your dataset into training and validation sets. Standardize features (zero mean, unit variance), especially important for L1 and Elastic Net.
• Step 2: Choose a base model. For regression, start with linear regression. For classification, consider logistic regression (which also supports L1/L2 penalties).
• Step 3: Pick a type of regularization. If you need feature selection, start with L1 (Lasso). If predictive accuracy and stability are priorities, start with L2 (Ridge). If features are correlated and you still want sparsity, try Elastic Net.
• Step 4: Select candidate lambda values. Create a grid of values (e.g., 1e-4, 1e-3, 1e-2, 1e-1, 1, 10). Plan to try each and compare validation performance.
• Step 5: Train the model for each lambda. Fit the model using the training data and compute validation error (MSE for regression, accuracy or log loss for classification). Record performance and model characteristics (like number of nonzero weights for L1/Elastic Net).
• Step 6: Compare and choose lambda. Look for the value that minimizes validation error and avoids extreme underfitting or overfitting. Prefer simpler models when validation performance is similar (the simpler one is often more robust).
• Step 7: Retrain on combined training+validation data (optional). Once the best lambda is selected, retrain using more data for a final model. Then evaluate on a held-out test set to estimate true performance.
• Step 8: Interpret and deploy. With L1/Elastic Net, examine which features remain; with L2, note the shrinkage pattern. Ensure the preprocessing pipeline (e.g., standardization) is carried to production.

Tips and Warnings

• Start Simple: Begin with L2 for stability. If you also want interpretability and feature selection, try L1 or Elastic Net.
• Watch Lambda Size: Too large λ causes underfitting (high bias) with tiny weights; too small λ can overfit (high variance). Plot validation error vs. λ to see the U-shaped curve.
• Scaling Matters: Without feature scaling, L1 may prefer features just because of units. Standardization ensures fair penalty across features.
• Interpretation: L1 zeros out weights, but remember this depends on data and preprocessing. Do not assume causality from selection; it shows predictive usefulness, not cause-effect.
• Numerical Stability: L2 aids stability by making (X^T X + λI) invertible and better-conditioned. This helps when features are nearly collinear.
• Optimization Details: L1 introduces non-differentiable points at zero; use algorithms that handle this (coordinate descent, proximal methods). L2 is smooth and friendly for standard gradient methods.
• Evaluate on Unseen Data: Always assess regularized models on validation/test sets to ensure real generalization gains. Training loss alone can mislead.
• Combine with Other Practices: Use proper train/validation splits and, if needed, cross-validation to pick λ more reliably. Regularization complements, not replaces, good evaluation.

Deepening the Geometric Intuition

• Loss Contours: For two weights, the unregularized MSE loss forms ellipses centered at the least-squares solution. Moving along these ellipses increases loss.
• Constraint Sets: L2’s circle (or hypersphere) and L1’s diamond (or cross-polytope) define allowed regions for weights. The best constrained solution lies where an ellipse first touches the boundary (tangency).
• Why L2 Shrinks Smoothly: The circle has no sharp corners, so the tangency is unlikely to land exactly on an axis. Most weights are nonzero but reduced in magnitude.
• Why L1 Zeros Out: The diamond has sharp corners aligned with axes. Ellipses often touch corners, forcing one or more weights to be exactly zero. This is the geometric source of sparsity.

Extending Beyond Linear Regression (Conceptual)

• Logistic Regression: Replace MSE with logistic loss; add L1 or L2 penalties the same way. L1 still encourages sparsity; L2 still encourages smooth shrinkage.
• Other Models: Neural networks, support vector machines, and many others support L2-like penalties (often called weight decay). The philosophy remains: limit weight size to control complexity.

Putting It All Together

• Problem: Overfitting hurts generalization. Solution: Add regularization to loss to penalize large weights.
• Choices: L2 (Ridge) for stable shrinkage; L1 (Lasso) for sparsity and feature selection; Elastic Net to blend both. Control with lambda, chosen as a hyperparameter using validation.
• Intuition: Circle vs. diamond constraints explain the different behaviors. Tangency shows how the solution changes as λ varies.
• Practice: Standardize features, tune λ carefully, and evaluate on unseen data. Prefer simpler models when performance is similar for robustness.

This lecture presents regularization as a simple and essential tool for fighting overfitting. By adding a penalty to the cost function, regularization discourages large weights and thus overly complex models. Two main forms are emphasized: L2 (Ridge), which smoothly shrinks all weights toward zero without usually making them exactly zero, and L1 (Lasso), which often pushes some weights to be exactly zero, performing feature selection. Elastic Net combines both, balancing sparsity and stability, especially when features are correlated. The geometric view—loss ellipses with a circle (L2) or diamond (L1) constraint—explains why L2 yields shrinkage and L1 yields sparsity: the best point is where the lowest loss contour is tangent to the constraint.

In practice, you will tune the hyperparameter lambda to control the penalty’s strength. A small lambda may overfit, while a large one may underfit. The right lambda provides the best trade-off between fitting the data and keeping the model simple. Using validation methods to select lambda is standard practice. L2 is usually favored for strong predictive performance and stability; L1 is valuable when you want interpretability and feature selection.

To practice, start with a basic linear regression and add L2 regularization; observe how training and validation errors change as you increase lambda. Then repeat with L1 and note which features remain. Try Elastic Net when you have many correlated features, adjusting the mix between L1 and L2. Make sure to standardize features, especially when using L1 or Elastic Net, to ensure fair treatment by the penalty.

Next steps include learning formal validation techniques to pick lambda more reliably and exploring regularization in other models such as logistic regression and neural networks. As you build more complex systems, remember the lecture’s core message: regularization is a principled way to control complexity. It adds a gentle brake to your model so it rides smoothly over new data instead of wobbling after every bump. Keep the circle and diamond picture in mind—it is a powerful mental model that explains the behavior of L2 and L1 in a single glance.