Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 7 - Agentic LLMs

This lecture teaches L1 regularization (also called LASSO), a powerful technique to prevent overfitting by adding a penalty to a model’s loss function that depends on the sum of absolute values of the model’s weights. Overfitting happens when a model fits training data extremely well but performs poorly on new, unseen data. The key idea behind regularization is to balance two goals at once: fit the data well and keep the model simple. L1 regularization creates simplicity by pushing many weights to be exactly zero, which automatically removes unhelpful features and makes the model easier to interpret.

The lecture starts by reminding you why we regularize at all: models with too many degrees of freedom can latch onto noise in the dataset. To fight this, we add a penalty to the loss, giving the objective function: objective = loss + λ × penalty. The parameter λ (lambda) controls how strong the penalty is; higher λ means stronger regularization, which usually shrinks or zeroes more coefficients. The lecture then contrasts two main types of penalties: L2 (Ridge), which uses the sum of squared weights, and L1 (LASSO), which uses the sum of absolute values. This difference might look small at first, but it leads to very different outcomes when optimizing.

A central part of the lecture is the geometric intuition. In two dimensions with weights w1 and w2, you can imagine the optimization as moving contour lines (like elevation lines on a map) of the loss function until they just touch a “constraint region.” For L2, the constraint region is a circle (because w1^2 + w2^2 ≤ c). For L1, the constraint region is a diamond (because |w1| + |w2| ≤ c). Circles are smooth, while diamonds have sharp corners that sit right on the axes. When the loss contours press against the diamond, they are more likely to touch at a corner, which makes one weight exactly zero. This geometric picture explains why L1 creates sparse solutions (lots of zeros), while L2 rarely does.

The lecture then discusses when to use each method. If you have many features and suspect many are irrelevant (common in high-dimensional problems), L1 is very useful because it performs feature selection automatically. If you have a moderate number of features and believe most matter at least a little, L2 is often better, as it shrinks all weights toward zero without turning them off completely. Sometimes you might want both effects: some sparsity and some stability. That is where Elastic Net comes in—a combined penalty that blends L1 and L2, controlled by a mixing hyperparameter.

Next, the talk covers practical pros and cons. Advantages of L1 include built-in feature selection, which leads to simpler, more interpretable models with fewer non-zero coefficients. This can also reduce storage and computation for prediction. Disadvantages include computational considerations: the absolute value function in L1 is not differentiable at zero (it has a sharp corner), so basic gradient descent doesn’t apply directly. Instead, you use methods like subgradient descent or coordinate descent, which can be slower than the smooth-gradient methods commonly used for L2. Another drawback is stability: L1 solutions can change a lot if the data changes slightly, especially when features are highly correlated, because different features may take turns being selected.

The instructor also answers common questions. Why is L1 computationally heavier? Because its penalty is non-differentiable at zero, forcing us to use slower methods like subgradient or coordinate descent. Why not just remove irrelevant features before training? You often don’t know which ones are irrelevant, or they may have tiny effects that are still useful; L1 can discover this automatically and keep small but real signals if λ isn’t overly large. Finally, choosing λ is critical and is typically done by cross-validation: you try different λ values, check validation performance, and pick the best balance.

By the end, you should be able to: explain why we regularize, write the L1 objective function, describe how λ controls the trade-off between fit and simplicity, explain geometrically why L1 sets coefficients to zero while L2 does not, decide when to use L1 vs L2 vs Elastic Net, and understand the computational differences and algorithms used for L1. The lecture is designed for beginners to intermediate learners who know basic linear models and loss functions. Prior knowledge helpful here includes linear regression, the idea of fitting a loss function, and basic optimization concepts like gradients. With these ideas, you can apply L1 to build simpler, more interpretable, and better-generalizing models.

1. Overall Architecture/Structure of Regularized Learning

At a high level, supervised learning with regularization has these components:

• Data: Inputs X (features) and outputs y (targets).
• Model: A function f(x; w) with parameters w (e.g., linear regression f(x) = w·x).
• Loss: A measure of prediction error (e.g., mean squared error for regression).
• Regularizer: A penalty that discourages complex models by shrinking weights.
• Objective: The sum of the loss and the penalty, weighted by λ (lambda), the regularization strength.
• Optimizer: An algorithm to minimize the objective (e.g., subgradient or coordinate descent for L1).
• Validation: A method to select λ and compare models (e.g., cross-validation).

Data flow: You start with data (X, y). You choose a model and define the loss. You add a regularization term. You minimize the objective to find weights w*. During optimization, the regularizer influences w* by shrinking or zeroing coefficients. Finally, you evaluate on held-out data to ensure generalization.

1. L1 Objective Function and Constrained Form

For a linear regression model, the L1-regularized objective is: Objective(w) = Loss(y, Xw) + λ × Σ_i |w_i|.

• Loss(y, Xw) is often the mean squared error: (1/2n) Σ (y_j − x_j·w)^2, where j indexes data points.
• The L1 penalty is the L1 norm: ||w||_1 = Σ_i |w_i|.
• λ ≥ 0 controls the strength of the penalty.

Constrained view: Minimize Loss(y, Xw) subject to ||w||_1 ≤ c for some constant c. There is a one-to-one mapping between λ in the penalized problem and c in the constrained problem (for convex problems). The constrained view gives geometric insight:

• L1 constraint: |w1| + |w2| ≤ c forms a diamond in 2D.
• L2 constraint: w1^2 + w2^2 ≤ c forms a circle in 2D.

1. Geometric Intuition: Why L1 Creates Sparsity

Imagine the loss function’s contour lines (like height lines on a hill) around its unconstrained minimum. As you tighten the constraint (smaller c) or increase λ, the feasible region shrinks toward the origin. For L1, the diamond has corners on the axes. When the loss contours are pushed outward until they just touch the diamond, they often touch at a corner—exactly where one coordinate is zero. This makes the solution lie on an axis (some w_i = 0). In higher dimensions, the L1 feasible set is a polytope with many corners. Touch points at these corners naturally create many zeros, leading to sparsity.

In contrast, L2’s circle is smooth and symmetrical with no corners. When contours touch it, the solution tends to have all components non-zero, though they are small. This smoothness also makes L2 easy to optimize with standard gradient methods.

1. Optimization: Why L1 Needs Special Methods

The absolute value function |w| is not differentiable at w = 0; its left derivative is −1 and right derivative is +1. Gradient descent relies on the derivative to tell you which way to move. Because of the kink at zero, you cannot compute a unique gradient there. Two common methods handle this:

• Subgradient Descent: The subgradient generalizes the derivative for convex but non-smooth functions. For |w|, the subgradient at w ≠ 0 is sign(w), and at w = 0 it is any value in [−1, 1]. Subgradient descent takes small steps in a subgradient direction to reduce the objective. It works but can be slower to converge because directions are less precise than true gradients.

• Coordinate Descent: Instead of updating all coefficients at once, coordinate descent updates one coefficient at a time, holding the others fixed. For objectives with separable penalties like L1, each 1D subproblem has a simple structure. The update often applies a thresholding rule that either shrinks the coefficient or sets it exactly to zero, naturally producing sparsity. This method is computationally efficient for high-dimensional problems and is widely used in practice for L1-regularized linear models.

Note: Smooth L2 penalties allow fast methods (like standard gradient descent with line search or second-order methods) because the gradient is well-behaved everywhere. This is why L2 training is typically simpler and faster.

1. Practical Behavior: L1 vs L2 vs Elastic Net

• L1 (LASSO): Encourages sparsity by setting many weights to exactly zero. Great for feature selection, interpretability, and when only a few features truly matter. Can be unstable when features are highly correlated; slight data changes can swap which feature is chosen. Training requires non-smooth optimization methods.

• L2 (Ridge): Shrinks all weights smoothly toward zero but rarely makes them exactly zero. Good when all features have some predictive power, and when you prefer stability. Easy and fast to optimize with smooth methods.

• Elastic Net: Combines L1 and L2 penalties. This can select groups of correlated features rather than arbitrarily picking just one, improving stability. The mixing parameter α (between 0 and 1) controls how much L1 vs L2 you use, and λ controls overall strength.

1. Choosing λ (Regularization Strength)

λ controls the balance between fitting the training data and keeping the model simple. If λ is too small, the penalty is weak: the model may overfit. If λ is too large, too many weights go to zero (for L1) or become too small (for L2), causing underfitting. The standard approach is to pick λ by cross-validation:

• Split data into training and validation folds.
• Train models over a grid of λ values.
• Choose the λ that gives the best validation performance.

In practice, you may use a logarithmic grid for λ (e.g., 10^-4, 10^-3, …, 10^2) to explore a wide range. Some workflows also consider the “one-standard-error rule” to pick a simpler model with performance statistically close to the best.

1. Interpreting Results and Stability Considerations

L1’s sparsity makes interpretation straightforward: non-zero coefficients highlight important features. However, when features are correlated or redundant, L1 may pick one and drop the rest arbitrarily. This can make interpretations shift when the data changes slightly. To mitigate this:

• Consider Elastic Net to encourage group selection.
• Check stability across resamples or folds to see if the same features keep getting selected.
• Report sets of candidate features rather than a single definitive list when correlations are high.

1. Advantages and Disadvantages Summarized

Advantages of L1 (LASSO):

• Automatic feature selection (sparsity).
• Clearer interpretability (few non-zero coefficients).
• Potentially faster prediction at inference time (fewer active features).

Disadvantages of L1:

• Non-differentiable penalty requires specialized, sometimes slower optimization.
• Less stable under data perturbations, especially with correlated features.
• May discard weak but jointly useful features if λ is too large.

Advantages of L2 (Ridge):

• Smooth optimization with standard gradient methods.
• Stable coefficients that change gradually with data.
• Keeps all features, which can help when each carries small signal.

Disadvantages of L2:

• No exact zeros, so no automatic feature selection.
• Models can be harder to interpret with many small but non-zero coefficients.

1. Implementation Guide (Framework-Agnostic)

Step 1: Prepare data.

• Gather features X and target y. Handle missing values and ensure features are numeric.
• Optionally standardize features (mean 0, variance 1) so the penalty treats them comparably.

Step 2: Choose the model and loss.

• For regression, choose linear regression with mean squared error (MSE).
• For classification (logistic regression), the loss is logistic (cross-entropy), and L1 can also be applied similarly.

Step 3: Define the L1 objective.

• Objective(w) = Loss(y, Xw) + λ × Σ|w_i|.

Step 4: Select optimization method.

• Coordinate descent: Efficient for L1; iterate over features, update each weight, repeat until convergence.
• Subgradient descent: Use appropriate step sizes; can be slower.

Step 5: Tune λ via cross-validation.

• Create a grid of λ values. Train models for each λ and evaluate on validation folds.
• Pick the λ with best validation score (or the simplest within one standard error of the best).

Step 6: Fit final model.

• Retrain on the full training set using the chosen λ.
• Record which features are non-zero; these are your selected features.

Step 7: Evaluate on test data.

• Measure performance (e.g., RMSE for regression, accuracy/AUC for classification).
• Check calibration and residuals for regression to ensure no obvious patterns remain.

Step 8: Communicate results.

• Report selected features and their coefficients.
• Explain trade-offs, especially if sparsity caused some features to be dropped.

1. Tips and Warnings

• Standardize features: L1 penalizes all coefficients equally; without standardization, features on larger scales may be unfairly penalized less or more.
• Beware correlated features: L1 may pick one and drop others; consider Elastic Net when groups of features represent similar signals.
• Don’t over-penalize: Very large λ can produce an empty or nearly empty model; always validate.
• Check stability: Use bootstrap or repeated cross-validation to see if the same features keep appearing.
• Start simple: Try pure L1 and pure L2 first to understand behaviors, then try Elastic Net for a balance.
• Monitor convergence: For coordinate descent, ensure you iterate until changes in coefficients are tiny; for subgradient, use appropriate step-size schedules.
• Interpret carefully: Zero coefficients mean exclusion; small non-zero values under L2 may still be meaningful signals.

1. Real-World Use Cases

• High-dimensional text (bag-of-words or n-grams): L1 selects a compact vocabulary of informative terms.
• Genomics and proteomics: L1 narrows thousands of biological markers to a handful that predict outcomes.
• Click-through rate prediction: L1 picks the most predictive user or context features among many.
• Sensor fusion: When most sensors are noisy, L1 can isolate the few that carry strong signal.
• Risk modeling: Sparse models aid compliance and interpretability in regulated industries.

1. Summary of Core Mathematics

• Objective: L(w) + λ||w||_1, with L convex (e.g., MSE or logistic loss) and ||w||_1 = Σ|w_i|.
• Subgradient for |w_i|: sign(w_i) if w_i ≠ 0; any value in [−1, 1] if w_i = 0.
• Optimality intuition: If the gradient of the loss with respect to w_i is small in magnitude, L1 can set w_i to zero; if large, it can overcome the penalty and keep w_i non-zero.
• Geometry: L1 constraint is a cross-polytope (diamond in 2D); loss contours touch at corners → sparse solutions.
• Trade-off: Larger λ → higher bias, lower variance; smaller λ → lower bias, higher variance.

These technical details equip you to understand why L1 behaves differently from L2, how to optimize with L1’s non-smooth penalty, and how to apply, tune, and interpret models that use L1 in practice.

This lecture focused on L1 regularization (LASSO) as a tool to prevent overfitting by adding a penalty equal to the sum of absolute values of model weights. We contrasted L1 with L2 (Ridge), showing how L1 tends to set many coefficients exactly to zero, achieving feature selection, while L2 shrinks weights smoothly without making them zero. The geometric intuition clarified this: the L1 constraint is a diamond with sharp corners that often force solutions onto axes, whereas the L2 constraint is a smooth circle that rarely produces exact zeros. We covered the role of the hyperparameter λ, which balances fitting the data against keeping the model simple; selecting λ via cross-validation is essential. We also addressed the computational challenge: L1’s non-differentiability at zero requires methods like subgradient descent or coordinate descent, which can be slower than methods for L2.

Practically, use L1 when you expect many irrelevant features and want interpretability through sparsity. Use L2 when most features are somewhat useful and you want stable, smooth shrinkage. When features are correlated or you want both sparsity and stability, Elastic Net blends the two approaches. Keep in mind the potential instability of L1 when data changes slightly and consider remedies like Elastic Net and stability checks.

To solidify learning, try fitting simple linear models with and without L1 on a dataset with many features, and compare which features become zero as you vary λ. Plot validation error across a λ grid to observe the U-shaped curve and pick a good λ. Experiment with coordinate descent versus subgradient descent to see differences in convergence speed. For next steps, explore Elastic Net in deeper detail, learn more about optimization methods for non-smooth problems, and practice with real high-dimensional datasets such as text or genomics. The core message to remember: regularization is about wise restraint. L1’s special power is to trim models down to their most important pieces, delivering better generalization and clearer stories about which features matter.