Stanford CS230 | Autumn 2025 | Lecture 10: What’s Going On Inside My Model?

This lecture teaches the foundations of linear regression, a core model in supervised learning that predicts a numeric output from input features by assuming a straight-line (linear) relationship. You will learn the full pipeline: how to define the model, how to choose an objective function that measures error, and how to solve for the best weights mathematically. The lecture uses clean matrix notation to handle many data points and features at once, and it shows how calculus and linear algebra combine to produce a closed-form solution known as the normal equation. On top of this, you will see how to evaluate model quality using mean squared error and how to control overfitting with regularization (both L2/ridge and L1/lasso), including why L1 tends to create sparse, feature-selecting solutions.

The lecture starts by setting up the notation: X is the data matrix with n rows (one per data point) and d columns (one per feature), Y is the vector of outputs, and W is the vector of model parameters (one weight per feature). The linear model predicts each output as a weighted sum of features, written compactly as y = W^T x for a single example, or Y ≈ XW across the entire dataset. The objective is to find W that makes predictions close to true outputs. This closeness is measured by the Residual Sum of Squares (RSS), which adds up squared differences between predictions and actual values. Minimizing RSS gives the best-fitting line or hyperplane in the least-squares sense.

To minimize RSS, the lecture uses calculus: take the derivative of RSS with respect to W, set it to zero, and solve. Expanding the squared norm ||Y − XW||^2 and applying matrix calculus rules leads to the normal equation: W = (X^T X)^{-1} X^T Y. This formula gives the unique minimizer when X^T X is invertible (full rank). The lecture explains why invertibility can fail when there are more features than data points (d > n): then the columns of X are linearly dependent, so X^T X is not full rank and cannot be inverted, leading to no unique solution without additional constraints.

The lecture then turns to how we judge model performance using Mean Squared Error (MSE), which is simply RSS divided by n, the number of data points. MSE reports average squared error and is widely used because it is easy to understand and compare. Although multiplying by 1/n doesn’t change which W minimizes the objective, it does make the error scale interpretable and consistent across datasets of different sizes.

Next comes a central practical concern: overfitting. Especially with many features, linear regression can fit not only the true pattern but also random noise in the training data. The lecture presents regularization as the main defense: we add a penalty to discourage large weights, which smooths the solution and reduces variance. Two common choices are L2 (ridge), which penalizes the sum of squared weights, and L1 (lasso), which penalizes the sum of absolute values. A key insight is that L1 often yields sparse solutions by setting some weights exactly to zero, effectively selecting features. The geometric view helps: L2’s penalty contour is round and smooth, shrinking weights evenly, while L1’s diamond shape has sharp corners that tend to catch the solution at an axis, forcing some weights to be zero.

By the end of the lecture, you will understand: how to pose linear regression as minimizing RSS, how to derive and interpret the normal equation, why invertibility and rank matter, when overfitting appears, how regularization counters it, and how to evaluate with MSE. You will also be able to interpret the meaning of weights (positive increases prediction, negative decreases) and appreciate the trade-off controlled by the regularization strength λ. The material is foundational and prepares you for the next steps like logistic regression and more advanced models.

This lecture is aimed at beginners to early intermediates in machine learning who have basic familiarity with linear algebra (vectors, matrices, transpose, inverse) and calculus (derivative concepts). You should be comfortable with ideas like features and labels, and with reading equations involving matrices. After studying this material, you will be able to build and reason about linear regression models end-to-end, diagnose when the normal equation applies, decide when to add regularization, and use MSE to evaluate your model’s performance on data. The structure of the lecture flows from problem setup and notation, to model and objective, to optimization and solution, to performance evaluation, and finally to extensions via regularization, closing with intuitive geometric reasons for lasso’s sparsity.

1. Overall Architecture/Structure

• Data and Notation: We organize the dataset into a matrix X ∈ R^{n×d}, where n is the number of data points and d is the number of features. Each row X_i: is a 1×d feature vector for the i-th data point, and each column corresponds to one feature across all points. Outputs are collected in a vector Y ∈ R^{n×1}, and parameters (weights) are in W ∈ R^{d×1}. The model predicts Ŷ = XW.

• Model Family: Linear regression assumes the output is a linear function of inputs. For a single data point x ∈ R^d, the prediction is ŷ = W^T x. This is a weighted sum: each entry w_j tells how strongly feature j influences the prediction; sign indicates direction, magnitude indicates strength.

• Objective Function: We want W that makes predictions close to actual outputs. We measure closeness with Residual Sum of Squares (RSS): RSS(W) = ||Y − XW||2^2 = (Y − XW)^T (Y − XW) = ∑{i=1}^n (y_i − x_i^T W)^2. Squaring emphasizes larger errors and yields a smooth, convex objective that is easy to optimize.

• Optimization Strategy: Use calculus to minimize the differentiable objective. Take the derivative of RSS with respect to W, set it to zero, and solve the resulting linear system. This produces a closed-form answer (the normal equation) under an invertibility condition.

• Solution: The gradient of RSS(W) is ∇_W RSS = −2 X^T Y + 2 X^T X W. Setting ∇_W RSS = 0 gives X^T X W = X^T Y. If X^T X is invertible (full rank d), we solve W = (X^T X)^{-1} X^T Y. This W uniquely minimizes RSS because RSS is convex and quadratic in W.

• Invertibility and Rank: X^T X is invertible if and only if the columns of X are linearly independent. Linear independence means no column can be formed as a combination of the others. If d > n, there cannot be d independent columns in n-dimensional space, so dependence is guaranteed. In that case, multiple weight vectors yield the same predictions and RSS, so there’s no unique minimizer without further constraints.

• Evaluation Metric: Mean Squared Error (MSE) = (1/n) RSS = (1/n) ∑(y_i − x_i^T W)^2. Though dividing by n doesn’t change the optimizer, it makes the error interpretable as an average per example, useful for comparing across datasets or different n.

• Overfitting Risk: With many features or with near-linear dependencies, the model can fit noise in training data, making training error low but test error high. Overfitting is especially likely when d is large relative to n or when some features are redundant. Coefficients can become very large in magnitude to chase tiny variations, amplifying noise.

• Regularization: To curb overfitting and fix non-uniqueness, we adjust the objective to penalize large weights. Two common forms: L2 (ridge): minimize RSS + λ||W||_2^2; L1 (lasso): minimize RSS + λ||W||_1. The hyperparameter λ ≥ 0 controls penalty strength. Larger λ prefers smaller W, trading some bias for lower variance (more stable predictions).

• Geometry of Penalties: L2 penalty forms circular/spherical level sets in weight space, gently shrinking all coefficients toward zero. L1 penalty forms diamond-shaped level sets with sharp corners at axes; minimization often meets the objective at these corners, setting some coefficients exactly to zero. This geometric difference explains ridge’s smooth shrinkage versus lasso’s sparsity.

1. Code/Implementation Details (Conceptual, No Specific Language Required)

• Matrix Computations: The core operations are X^T X (d×d), X^T Y (d×1), and solving a d×d linear system. Numerically, computing (X^T X)^{-1} explicitly is less stable than solving the system X^T X W = X^T Y via a linear solver. However, conceptually, the closed-form inverse clarifies the solution.

• Derivative Steps in Detail: Let f(W) = ||Y − XW||_2^2 = (Y − XW)^T (Y − XW). Expand: f(W) = Y^T Y − 2 Y^T X W + W^T X^T X W. Take derivative w.r.t. W: ∇ f(W) = −2 X^T Y + 2 X^T X W. Set to zero: X^T X W = X^T Y ⇒ W = (X^T X)^{-1} X^T Y (if invertible).

• Conditions and Edge Cases: • If X^T X is not invertible (rank-deficient), there are infinitely many minimizers of RSS. Regularization with L2 makes the matrix X^T X + λI invertible for any λ > 0, yielding a unique solution. • With L1, the solution is not given by a simple matrix inverse; specialized optimization (e.g., coordinate descent) is typically used to find the sparse minimizer.

• Regularized Objective Derivatives (Intuition): • L2 (ridge): J(W) = ||Y − XW||^2 + λ||W||^2. The derivative adds 2λW, giving (X^T X + λI) W = X^T Y, so W = (X^T X + λI)^{-1} X^T Y. • L1 (lasso): J(W) = ||Y − XW||^2 + λ∑|w_j|. The absolute value is not differentiable at zero, creating the “kink” that induces sparsity. Solutions often lie at exact zeros for some coefficients, which is why lasso selects features.

1. Tools/Libraries Used

• The lecture stays at the mathematical level and does not depend on specific libraries. In practice, any scientific computing environment (such as NumPy, MATLAB, or R) can perform the needed matrix operations: transposes, multiplications, and solving linear systems. No special machine learning framework is required for basic linear regression with the normal equation.

1. Step-by-Step Implementation Guide (Conceptual)

• Step 1: Prepare Data • Collect features into X (n×d) and outputs into Y (n×1). Ensure each row of X matches the corresponding entry of Y. Consider adding a column of ones to X if you include an intercept (bias term); this was not explicitly discussed here, but it’s a common practice so the model can fit a nonzero baseline.

• Step 2: Define Objective • Use RSS(W) = ||Y − XW||^2 if you seek a pure least-squares fit. For evaluation or comparison, compute MSE = RSS/n.

• Step 3: Solve for W (Unregularized) • Compute X^T X and X^T Y. If X^T X is invertible, compute W = (X^T X)^{-1} X^T Y (or solve the linear system directly using a solver). This yields the unique least-squares minimizer.

• Step 4: Check Invertibility / Overfitting Risk • If d > n or features are collinear, X^T X may not be invertible or may be poorly conditioned. Expect instability or multiple solutions; move to regularization.

• Step 5: Add Regularization (If Needed) • Ridge (L2): Solve (X^T X + λI) W = X^T Y for a chosen λ ≥ 0. Larger λ shrinks coefficients more and improves stability when features are correlated or d > n. • Lasso (L1): Solve minimize ||Y − XW||^2 + λ||W||_1 using an appropriate algorithm (e.g., coordinate descent). Expect some weights to become exactly zero, simplifying the model.

• Step 6: Evaluate with MSE • Compute predictions Ŷ = XW on validation or test data and calculate MSE. Compare MSE across different settings (e.g., various λ) to choose the best trade-off.

• Step 7: Interpret Weights • Inspect signs and magnitudes: positive means increasing the feature increases prediction; negative means it decreases. Very large magnitudes may indicate overfitting or feature scaling issues.

1. Tips and Warnings

• When d > n or columns are nearly dependent, regularization is not optional—it’s essential for stability and uniqueness. Even mild ridge regularization can dramatically reduce variance and improve generalization.

• L1 vs L2 Choice: If you desire interpretability and feature selection, lasso is attractive due to sparsity. If you want to reduce variance smoothly without forcing zeros, ridge is a safer, more stable shrinker, especially when many features have small, real effects.

• Lambda Sensitivity: Start with a range of λ values from very small to moderately large and pick the one with lowest validation MSE. Too-small λ provides little protection; too-large λ oversmooths, increasing bias and underfitting.

• Numerical Stability: Although the formula uses a matrix inverse, in computation prefer solving linear systems or using decompositions (like QR) instead of explicitly inverting X^T X. This reduces numerical errors.

• Interpreting MSE: Remember MSE averages squared errors; large outliers can dominate. If you see very high MSE driven by a few points, investigate data quality or consider robust alternatives (outside this lecture’s scope).

• Feature Understanding: If a weight is zero under lasso, that feature is effectively unused—this can guide feature selection. If all weights shrink too close to zero with ridge or lasso, you may have set λ too high or features may be weak predictors.

This lecture built linear regression from the ground up, from notation to solution to safeguards against overfitting. We began by organizing data in matrix form (X for features, Y for outputs, W for weights) and defining the linear model ŷ = W^T x. We chose the Residual Sum of Squares (RSS) as the objective to measure total squared prediction error and used calculus to derive the normal equation, W = (X^T X)^{-1} X^T Y, the closed-form minimizer when X^T X is invertible. The lecture clarified why invertibility requires full column rank and why this fails when there are more features than data points (d > n), resulting in no unique solution without added constraints.

We then learned to evaluate model performance using Mean Squared Error (MSE), an average form of RSS that enables fair comparisons across datasets. Recognizing the risk of overfitting—especially with many or redundant features—we introduced regularization. L2 (ridge) adds a squared-weight penalty that smoothly shrinks coefficients, improves stability, and often resolves near-dependencies. L1 (lasso) adds an absolute-value penalty that frequently sets some weights exactly to zero, providing natural feature selection. A geometric view explained the different effects: L2’s smooth circular contours encourage small but nonzero weights, while L1’s diamond-shaped contours with sharp corners make zero weights likely.

To practice, start by fitting an unregularized linear model on a small dataset and verify the normal equation solution. Next, explore d > n or near-collinearity to witness non-uniqueness or instability, then add ridge regularization and observe how λ stabilizes the solution and affects MSE. Finally, try lasso on a dataset with many weak features and confirm that some coefficients become exactly zero—an exercise in feature selection.

For next steps, extend this understanding to logistic regression for classification problems and to generalized linear models. Study strategies for choosing λ effectively, such as cross-validation, and learn more about numerical stability and matrix decompositions for practical computing. The core message to remember is the three-part structure of supervised learning—model, objective, optimization—and how linear regression makes each piece explicit and solvable. With RSS as the objective and the normal equation as the optimizer’s result, you gain a precise, interpretable, and often surprisingly strong baseline model, made robust by regularization when needed.