Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning

This lecture teaches the core idea of regularization in machine learning: how to prevent overfitting by penalizing model complexity so that models generalize well to new data. The central message is simple but powerful: a model that fits every tiny bump in the training data—often shown as a function with many 'wiggles'—may look perfect on past data but fail on future data. Regularization fixes this by adding a penalty into the training objective that discourages overly large or numerous parameters, leading the learning algorithm to prefer simpler, smoother solutions.

The lecture focuses on two widely used forms of regularization: L2 regularization (also called Ridge regression) and L1 regularization (also called Lasso regression). L2 regularization adds a penalty based on the sum of squared weights, which encourages all weights to be small but rarely exactly zero. This acts like a gentle pull toward zero and results in smoother functions. L1 regularization adds a penalty based on the sum of absolute weight values, which encourages many weights to become exactly zero. This sparsity helps with feature selection by turning off unimportant features automatically. The lecture also introduces Elastic Net regularization, which combines L1 and L2 to get the benefits of both.

A practical question addressed here is how to choose the penalty strength, usually called lambda. The answer is to treat lambda as a hyperparameter and pick it using cross-validation. Specifically, you split your data into training, validation, and test sets. You train models with different lambda values on the training set, choose the one that performs best on the validation set, and finally report performance on the test set so you get an honest estimate of generalization. This workflow prevents sneaky overfitting to the test set.

Polynomial regression serves as an intuitive example of how complexity can harm generalization. Increasing the polynomial degree typically drives the training error down by capturing more patterns. However, after a point, the test error starts rising because the model starts fitting noise and making many sharp turns—the 'wiggles.' Regularization reduces the size of the coefficients, which calms down these wiggles and improves test performance.

The lecture emphasizes that regularization is even more important for neural networks because they often have a huge number of parameters. Methods such as L2/L1 penalties, dropout, and batch normalization help deep models generalize better. Dropout randomly turns off some neurons during training, which stops the network from depending too much on any one path and works like averaging many small networks. Batch normalization normalizes intermediate activations and tends to stabilize training and indirectly regularize the network.

Another key perspective presented is the Bayesian interpretation. In Bayesian terms, regularization encodes your prior belief about what parameter values are likely before seeing data. L2 corresponds to a Gaussian (normal) prior that prefers small weights, while L1 corresponds to a Laplace prior that strongly prefers many exact zeros. This view gives a theoretical justification for why these penalties make sense.

By the end, you understand what regularization is, why it’s necessary, and how to apply L1, L2, and Elastic Net. You also know how to choose lambda using cross-validation and how these ideas extend naturally to neural networks. The flow of the lecture moves from the motivation (overfitting and wiggly functions), to concrete tools (L1/L2/Elastic Net), to practical model selection (cross-validation), to broader applications (neural networks), and finally to theory (Bayesian priors).

1. Overall Architecture/Structure of Regularized Learning

• Goal (What are we solving?): We are solving supervised learning problems (like regression or classification) where we want good predictions on new, unseen data. The core challenge is overfitting—when a model fits training data too closely and performs poorly on test data.
• Basic Idea (How do we address it?): We add a penalty term to the loss function that punishes model complexity. Complexity is often measured by the size of the parameters (weights). The modified training objective becomes: Loss = Prediction Error + Regularization Penalty.
• Components: (a) Data split: training, validation, test. (b) Model: linear regression, polynomial regression, neural networks, etc. (c) Loss function: measures how well predictions match targets (e.g., sum of squared errors). (d) Regularization: L2 (sum of squared weights) or L1 (sum of absolute weights), or a mix (Elastic Net). (e) Hyperparameter lambda: controls how strong the penalty is. (f) Optimizer: algorithm to find parameters that minimize the total loss.
• Data Flow: 1) Choose a model family. 2) Define loss with regularization. 3) Train on the training set to minimize this loss. 4) Evaluate models with different lambda values on the validation set. 5) Pick the best lambda. 6) Retrain on train + validation (optional) with the chosen lambda. 7) Evaluate once on the test set for the final performance.

1. L2 Regularization (Ridge) in Detail

• Intuition: L2 encourages all weights to be small but not exactly zero. This smooths predictions, reduces sensitivity to noise, and prevents extreme parameter values.
• Formal Description in Words: We add a penalty equal to lambda multiplied by the sum of squares of all weights. For a linear model y_hat = w^T x, the training objective becomes: sum of squared residuals plus lambda times sum of squared weights. The intercept (bias) term is often excluded from penalization so the baseline prediction remains flexible.
• Effects: (a) Shrinkage—coefficients are pulled toward zero. (b) Stability—helps when features are correlated by distributing weight among them. (c) Bias-Variance Trade-Off—adds some bias but reduces variance, usually improving test error.
• Practical Behavior: If lambda is tiny, the model acts like an unregularized one and can overfit. If lambda is huge, the model underfits because most weights are forced near zero.
• Optimization: The total loss is smooth and convex for linear regression, so it has a unique global minimum. Many solvers (closed-form matrix solution for Ridge; gradient-based methods) can compute it efficiently.

1. L1 Regularization (Lasso) in Detail

• Intuition: L1 encourages many weights to be exactly zero, creating a sparse model. This naturally performs feature selection.
• Formal Description in Words: Add lambda times the sum of absolute values of the weights to the prediction error. This creates 'kinks' in the loss surface at zero, which makes some weights settle exactly at zero.
• Effects: (a) Sparsity—turns off unhelpful features. (b) Interpretability—fewer nonzero coefficients make the model easier to understand. (c) Guard against overfitting—eliminates noisy signals.
• Practical Behavior: L1 can be unstable with highly correlated features because it may pick one and drop others arbitrarily. In such cases, Elastic Net can be better.
• Optimization: The objective is convex but not smooth at zero because of absolute values. Specialized solvers (like coordinate descent) handle it well.

1. Elastic Net Regularization

• What It Is: A combination of L1 and L2 that blends sparsity and stability. The loss includes lambda1 times absolute weights plus lambda2 times squared weights.
• Why Use It: When you want feature selection but also want to avoid L1’s tendency to pick a single feature among a group of correlated features. L2 helps share weights among related features.
• Behavior: By adjusting the two lambdas (or a total lambda with a mixing ratio), you can move between mostly-L1 behavior and mostly-L2 behavior.
• Optimization: Still convex for linear models. Coordinate descent works well in practice.

1. Choosing Lambda via Cross-Validation

• Hyperparameter Tuning: Lambda is not learned directly from training loss; it must be selected using a validation procedure. Use a grid (e.g., powers of 10) or a finer search over promising ranges.
• Data Splitting: Keep a clean separation: training for fitting parameters; validation for choosing lambda; test for final performance only. Do not use test data to make any choices.
• K-Fold Cross-Validation: Instead of a single validation set, you can split training data into K folds. Train on K-1 folds and validate on the remaining fold, rotating so each fold is used once as validation. Average performance across folds to select lambda.
• Practical Tips: Start with a wide search (e.g., lambda ∈ {1e-4,...,1e4}). If the best lambda is at a boundary, expand the search range.

1. Polynomial Regression Example Mechanics

• Model: y_hat = w0 + w1x + w2x^2 + ... + wd*x^d. Higher degree d increases flexibility.
• Without Regularization: As d grows, training error shrinks drastically, but the curve becomes wiggly and test error rises after a point. Large coefficients for high powers cause sharp bends.
• With L2: Penalizes large coefficients evenly, reducing severe bends and improving generalization. Coefficients are shrunk but typically not zeroed.
• With L1: Can completely drop some polynomial terms (e.g., set w7 = 0), yielding a simpler polynomial. Good when many powers are unnecessary.

1. Regularization in Neural Networks

• Why Critical: Neural networks have many parameters and can memorize training data easily.
• L2/L1 in NNs: Add penalties on all layer weights to discourage overly large values. Many deep learning frameworks call L2 'weight decay' because weights decay toward zero during training.
• Dropout: Randomly zeros a fraction (e.g., 0.5) of neuron outputs during training. This forces the network to distribute learning and reduces co-adaptation of neurons.
• Batch Normalization: Normalizes activations within a minibatch to stabilize training. Although primarily for optimization stability, it often improves generalization as a side effect.
• Implementation Notes: Apply L2/weight decay via optimizer settings; insert dropout layers after activations; include batch normalization layers near the start of each block.

1. Bayesian Interpretation

• Prior Beliefs: Before seeing data, assume parameters are likely small (L2) or many are exactly zero (L1). Data updates these beliefs to produce the final parameters (the posterior).
• L2 ↔ Gaussian Prior: Prefers small weights around zero, with probability decreasing smoothly as weights grow. This leads naturally to the squared penalty.
• L1 ↔ Laplace Prior: Has a sharp peak at zero and heavier tails, encouraging many exact zeros but allowing some larger weights. This leads naturally to the absolute value penalty.
• Why This Matters: Provides a principled, theoretical reason for penalties and helps understand how assumptions translate into model behavior.

1. Practical Implementation Guide Step 1: Split Data

• Divide data into training, validation, and test sets (e.g., 60/20/20). Ensure the test set is only used once at the end.

Step 2: Choose a Model Family

• For tabular regression: start with linear or polynomial regression. For high-dimensional or noisy data, consider regularized linear models. For complex patterns, consider neural networks with regularization.

Step 3: Define Loss with Regularization

• L2: Loss = squared error + lambda * sum of squared weights. L1: Loss = squared error + lambda * sum of absolute weights. Elastic Net: mix both with their own strengths.

Step 4: Tune Lambda

• Prepare a candidate set (e.g., log-spaced values). Train the model for each lambda on the training set and evaluate on the validation set or via K-fold CV.

Step 5: Pick and Retrain

• Select the lambda giving the best validation performance. Optionally retrain on combined train+validation data using this lambda.

Step 6: Final Test

• Evaluate once on the test set to estimate real-world performance.

1. Tools/Libraries (Conceptual Usage)

• Scikit-learn (Python): Ridge, Lasso, ElasticNet, and CV versions like RidgeCV, LassoCV, ElasticNetCV make tuning easier. They handle scaling and coordinate descent internally.
• Deep Learning Frameworks: PyTorch/Keras/TensorFlow support weight decay (L2), dropout layers, and batch normalization layers directly in model definitions and optimizers.

1. Tips and Warnings

• Don’t tune on the test set: This leaks information and inflates results. Always use a validation set or cross-validation for hyperparameters.
• Start with L2: If you don’t need feature selection, L2 is a safe default for stability and smoothness. Try L1 or Elastic Net when you suspect many irrelevant features.
• Check for underfitting: If validation error is high and training error is also high, lambda may be too large. Reduce lambda to allow more flexibility.
• Check for overfitting: If training error is low but validation error is high, lambda may be too small. Increase lambda to penalize complexity.
• Neural Networks: Use a combination—weight decay plus dropout (and often batch norm). Tune dropout rate (e.g., 0.1–0.5) and weight decay carefully.

1. Worked Walkthrough (Conceptual)

• Example Goal: Predict house prices from features like size, rooms, and age.
• Baseline: Train plain linear regression and observe overfitting if many features are noisy.
• Apply Ridge: Add L2 penalty; try lambda in {1e-4, 1e-3, ..., 1e2}. Pick the best by validation error; expect smoother, more stable coefficients.
• Apply Lasso: Try L1 for feature selection; many coefficients may become zero, simplifying the model.
• Elastic Net: If features are correlated (e.g., size and square footage), Elastic Net can share weights and avoid dropping useful features entirely.

1. Understanding the Trade-Off

• Training vs Generalization: Regularization slightly increases training error but often reduces validation/test error. The penalty trades a bit of fit for more reliability on new data.
• Smoother Functions: Especially visible in polynomial regression, where regularization reduces wild oscillations. Smoother functions are less sensitive to tiny data changes.
• Feature Pruning (L1): When many features exist, L1 helps automatically remove the unhelpful ones. This often improves interpretability and reduces variance.

1. Connecting to Theory

• Convexity: For linear models with L1/L2 penalties, the optimization problems are convex, ensuring a global optimum. This makes solutions stable and reliable.
• Bayesian Grounding: Priors explain why penalties work and how they encode assumptions. This perspective aligns practical tuning with theoretical understanding.

1. Extending to Classification

• Although the lecture focuses on regression language, the same penalties apply to classification models (e.g., logistic regression). You simply use a different prediction error term (e.g., log loss) but keep the same L1/L2 penalties. The logic of lambda tuning and cross-validation remains the same. This generality makes regularization a universal tool.

Regularization is a cornerstone of building models that truly generalize. The main idea is simple: add a penalty for model complexity so the learning algorithm prefers simpler, smoother solutions. L2 regularization (Ridge) shrinks weights toward zero, improving stability without usually zeroing them out. L1 regularization (Lasso) pushes many weights to exactly zero, providing sparsity and automatic feature selection. Elastic Net blends both, offering a practical balance when features are correlated or when you want both sparsity and stability.

The choice of penalty strength, lambda, is critical and must be tuned using cross-validation. By splitting data into training, validation, and test sets—and only using the test set once at the very end—you ensure an honest estimate of how the model will perform in the real world. Polynomial regression offers an intuitive picture of overfitting: as flexibility grows, training error can drop while test error rises. Regularization counters this by shrinking large coefficients and smoothing away the wiggles.

For neural networks, regularization is even more important because of their large number of parameters. L2/L1 weight penalties, dropout, and batch normalization help these models avoid memorizing noise and instead learn patterns that transfer to new data. Finally, the Bayesian perspective connects practice to theory by showing that regularization encodes prior beliefs about parameter sizes and sparsity.

Right after this, you can apply L2, L1, or Elastic Net to your own regression or classification tasks, tune lambda via cross-validation, and check how your test performance improves. If you work with deep learning, add weight decay and dropout, and consider batch normalization to stabilize training. The core message to remember is this: controlling complexity through regularization is essential for building reliable machine learning models that perform well beyond the training data.