Stanford CS230 | Autumn 2025 | Lecture 3: Full Cycle of a DL project

This lecture teaches the core ideas behind supervised learning for regression, focusing on the classic and foundational method: linear regression. In supervised learning, you are given input–output pairs: each input x is a vector of features (for example, house characteristics like square footage, number of bedrooms, and location), and each output y is the real number you want to predict (such as the house price). The goal is to learn a function f that maps any new input x to a good prediction of y. Linear regression assumes this mapping is a straight line (technically, a hyperplane) in feature space: f(x) = w^T x + b, where w is a vector of weights and b is a bias (intercept).

To measure how well the model predicts, the lecture uses Mean Squared Error (MSE), which averages the squares of the differences between actual values and predicted values. Squaring emphasizes larger mistakes more than smaller ones, which helps the optimization focus on big errors. The parameters w and b are found by minimizing MSE. The tool for this is gradient descent, an iterative algorithm: start with an initial guess for w and b, compute the gradient (the direction that increases the loss fastest), and step in the opposite direction to reduce the loss. The size of each step is determined by a learning rate hyperparameter.

The lecture explains that choosing a good learning rate is essential. If it is too large, the updates can overshoot the minimum, causing the loss to bounce or even explode. If it is too small, progress is painfully slow and training takes a long time. Helpful strategies include starting small and increasing until improvement slows, then backing off, using learning rate schedules that gradually reduce the rate during training, or using adaptive methods like Adam that adjust the effective learning rate for each parameter automatically.

Linear regression is praised for being simple, interpretable, and computationally efficient. It is often recommended as a baseline: try it first, and if it performs sufficiently well, you may not need a more complex model. However, it has limitations. The most important one is that it assumes a linear relationship between features and the target, which is not always realistic. The lecture gives examples of non-linear relationships: electricity demand versus temperature tends to be high at both hot and cold extremes, which looks like a U-shaped curve rather than a straight line; tree height versus age shows rapid growth early and slower growth later, another clearly non-linear pattern. Linear regression is also sensitive to outliers due to the squared-error loss, and it can overfit when it memorizes training noise rather than learning general patterns.

To address overfitting, regularization adds a penalty to the loss that discourages overly complex models. The lecture presents two main types: L1 regularization (lasso) adds the sum of absolute values of the weights to the loss, promoting sparsity (many weights become exactly zero) and thus performing feature selection. L2 regularization (ridge) adds the sum of squared weights, shrinking weights toward zero but rarely turning them off completely, which stabilizes the fit and reduces variance. A hyperparameter lambda controls the strength of this penalty: higher lambda means a stronger push toward simpler models, while lower lambda allows more flexibility.

The lecture also introduces polynomial regression as an extension that can capture non-linear relationships without abandoning linear regression’s simplicity. The trick is to expand the input features by adding polynomial terms (e.g., x^2, x^3), and then apply standard linear regression on the expanded feature set. This keeps the model linear in the parameters but allows curved fits. Ridge regression (linear regression with L2 regularization) and lasso regression (linear regression with L1 regularization) are highlighted as practical tools to control complexity and improve generalization.

By the end, you understand the full loop of building a basic regression model: define the model form (linear), choose a loss (MSE), minimize it with gradient descent (tuning the learning rate), be aware of outliers and non-linearity, and use regularization to prevent overfitting. With these ideas in place, you are set up to move on to logistic regression for classification problems, where many of the same optimization and hyperparameter principles apply. The lecture’s structure flows from the problem setup (supervised regression), to the model and loss, to optimization via gradient descent, to practical concerns (learning rate, pros/cons, outliers), and finally to extensions (regularization and polynomial features), giving a complete and clear foundation for predictive modeling with linear methods.

Overall Architecture/Structure

1. Problem Setup

• You have a dataset of n training examples. Each example has a feature vector x_i ∈ R^d (d features) and a real-valued target y_i ∈ R.
• The goal is to learn a function f(x) that predicts y for new, unseen inputs x.
• In linear regression, f(x) = w^T x + b, where w ∈ R^d (weights) and b ∈ R (bias) are the model parameters you must learn from data.

1. Loss Function: Mean Squared Error (MSE)

• Define predictions: for each i, ŷ_i = w^T x_i + b.
• The MSE loss is L(w, b) = (1/n) Σ_{i=1..n} (y_i − ŷ_i)^2.
• Squaring the errors penalizes large deviations more heavily than small ones, making the optimization focus on outliers’ impact (a strength and a weakness).

1. Optimization: Gradient Descent

• We minimize L(w, b) iteratively by following the negative gradient of the loss.
• Gradients for MSE (derivations): • ∂L/∂w = −(2/n) Σ x_i (y_i − ŷ_i) = (2/n) X^T (Xw + b1 − y), where X is the n×d matrix of features, y is the n-vector of targets, and 1 is the all-ones vector. • ∂L/∂b = −(2/n) Σ (y_i − ŷ_i) = (2/n) Σ (ŷ_i − y_i) = (2/n) 1^T (Xw + b1 − y).
• Update rules with learning rate α: • w ← w − α ∂L/∂w • b ← b − α ∂L/∂b
• Repeat until convergence (e.g., until loss reduction is below a small threshold for several iterations, or a maximum iteration count is reached).

1. Regularization (to prevent overfitting)

• L1 regularization (lasso): L(w, b) = (1/n) Σ (y_i − ŷ_i)^2 + λ Σ |w_j|.
• L2 regularization (ridge): L(w, b) = (1/n) Σ (y_i − ŷ_i)^2 + λ Σ w_j^2.
• Effects on gradients: • L2 adds 2λw to ∂L/∂w (for the common convention without 1/2 factors). The bias b is often not regularized. • L1 adds λ sign(w_j) to each component’s gradient. At w_j = 0, the subgradient is any value in [−λ, λ]; optimization uses subgradient or specialized solvers.
• λ (lambda) is a hyperparameter controlling penalty strength: higher values shrink more strongly.

1. Extensions: Polynomial Regression

• Expand x with polynomial features: for a scalar x, use [x, x^2, x^3, …]; for vector x, include squares and cross-terms if desired (though cross-terms were not emphasized in the lecture).
• Train the same linear regression on the expanded feature vector. The model remains linear in parameters but now can fit curves.

Data Flow

• Input: Matrix X (n×d) and vector y (n×1).
• Forward pass: Compute predictions ŷ = Xw + b·1.
• Loss computation: MSE between ŷ and y; optionally add regularization.
• Backward pass: Compute gradients w.r.t. w and b (plus regularization terms if used).
• Parameter update: Apply gradient descent steps; repeat for multiple iterations.

Code/Implementation Details (Conceptual Pseudocode) Language: Python with NumPy (conceptually)

1. Ordinary Linear Regression (MSE + Gradient Descent)

• Initialize w = zeros(d), b = 0 (or small random values).
• For t in 1..T: • y_hat = X @ w + b • error = y_hat − y • grad_w = (2/n) * X.T @ error • grad_b = (2/n) * np.sum(error) • w = w − α * grad_w • b = b − α * grad_b • Optionally record loss: (1/n) * np.sum(error**2)
• Stop early if loss stops improving materially.

1. With L2 Regularization (Ridge)

• Same as above, but add regularization to gradient: • grad_w = (2/n) * X.T @ error + 2λ * w • grad_b unchanged (commonly not regularized)

1. With L1 Regularization (Lasso)

• Replace grad_w by subgradient: • grad_w = (2/n) * X.T @ error + λ * sign(w) (componentwise) • Handle w_j = 0 carefully (subgradient ∈ [−λ, λ]); in practice, use proximal gradient (soft-thresholding) or coordinate descent for stability.

1. Polynomial Regression

• Create expanded feature matrix Φ: • For degree p: Φ = [x, x^2, …, x^p] (if x is 1-D), or augment columns accordingly if x is multi-dimensional. • Then run the same regression loop on Φ and y.

Important Parameters and Their Meanings

• α (learning rate): Step size for gradient descent. Too large causes divergence; too small leads to slow convergence.
• λ (regularization strength): Controls penalty weight. Higher values reduce variance but can increase bias.
• T (max iterations or epochs): Upper limit to guard against endless loops.
• Initialization: Starting values for w and b; zeros or small random values are common.

Code Execution Order and Flow

1. Prepare data: X, y; optionally standardize features if scales differ a lot (helps stability, though the lecture did not require it).
2. Initialize parameters w, b.
3. Loop: forward pass → loss → backward pass (gradients) → parameter update.
4. Check stopping conditions; possibly adjust α (learning rate schedule) if loss plateaus.
5. Output the learned w, b and evaluate predictions on new data.

Tools/Libraries Used (Common Choices)

• NumPy: For matrix operations (X @ w), sums, and vectorized computations.
• Optional: scikit-learn provides ready-made LinearRegression, Ridge, and Lasso implementations; PolynomialFeatures helps create polynomial inputs. While not part of the lecture’s code, these tools mirror the discussed techniques and are standard in practice.

Step-by-Step Implementation Guide Step 1: Define the task

• Choose a regression problem (e.g., predict house price). Identify features (inputs) and the target (output).

Step 2: Collect and prepare data

• Build your feature matrix X (n×d) and target vector y (n×1). Ensure no missing values in the simplest setup; remove or impute if present.
• Optionally remove obvious outliers or winsorize extreme values (because MSE is sensitive to outliers).

Step 3: Initialize parameters and hyperparameters

• Pick α (e.g., 0.01) and set λ if using regularization (start with small like 0.01). Initialize w (zeros) and b (0).

Step 4: Train with gradient descent

• Repeat: ŷ = Xw + b; compute loss; compute gradients; update w, b.
• Monitor loss to ensure it decreases steadily. If it stalls or oscillates, adjust α.

Step 5: Tune learning rate

• Try small α first; if loss decreases smoothly, you can consider slightly increasing. If it spikes or oscillates, reduce α immediately.
• Consider a schedule (e.g., reduce α by half after certain iterations without improvement) or use adaptive methods like Adam.

Step 6: Add regularization if overfitting

• For ridge: add 2λw to the gradient. For lasso: use subgradient or a proximal step to encourage sparsity.
• Adjust λ: increase if overfitting, decrease if underfitting.

Step 7: Extend with polynomial features if non-linearity is evident

• Add x^2 (and possibly x^3) terms to features. Retrain and re-tune λ (polynomial features can increase overfitting risk).

Step 8: Evaluate on new inputs

• Use f(x) = w^T x + b for predictions. Compare to known values when available to estimate generalization quality.

Tips and Warnings

• Learning rate pitfalls: Too large causes divergence (loss shoots up or becomes NaN); too small wastes time. Adjust promptly based on loss behavior.
• Outlier sensitivity: Because errors are squared, a few extreme points can dominate the fit. Consider outlier detection or robust preprocessing.
• Regularization choice: L1 is good for feature selection and sparse models; L2 stabilizes and spreads shrinkage. λ must be tuned for balance.
• Feature scaling: While not explicitly required in the lecture, scaling features can help gradient descent converge more smoothly, especially when feature magnitudes differ widely.
• Bias handling: Typically, do not regularize b; regularization focuses on limiting feature weights.
• Stopping criteria: Use thresholds on loss decrease or gradient norm to avoid unnecessary extra iterations.
• Non-linearity cues: U-shapes, saturations, or bends in scatter plots suggest adding polynomial features or considering non-linear models.
• Initialization: Zeros are fine for linear regression; complex initializations are unnecessary here.
• Monitoring: Plot training loss over iterations to visually confirm steady descent; sudden spikes indicate α is too high.

Why These Pieces Fit Together

• The linear model defines how predictions are made from parameters. The MSE loss defines what it means to be “good.” Gradients tell us how to change parameters to improve. The learning rate determines how fast (and safely) we change. Regularization ensures the model doesn’t just memorize training quirks. Polynomial expansion gives flexibility to capture curves while staying within the linear-in-parameters framework. Combined, they form a practical, end-to-end recipe for building strong baseline regressors.

This lecture laid out a complete, practical path to building a regression model that predicts real numbers from input features. You started with supervised learning, where labeled pairs (x, y) train a function to generalize to new inputs. Linear regression provided the foundational model f(x) = w^T x + b, and Mean Squared Error gave a clear objective to minimize. Gradient descent supplied the iterative method for tuning weights and bias, with the learning rate as a crucial hyperparameter controlling the speed and stability of learning.

You learned that linear regression is simple, fast, and interpretable—often the best baseline to try first. But it assumes linearity, is sensitive to outliers, and can overfit, especially when data is noisy or complex. Regularization addresses overfitting by penalizing large weights: L2 (ridge) shrinks them smoothly to stabilize the model, while L1 (lasso) drives many to zero, performing feature selection. The hyperparameter lambda governs how aggressively the model is simplified. To account for curved relationships, polynomial regression expands the feature set with powers like x^2, allowing the method to capture U-shapes and slowdowns without abandoning linear-in-parameter simplicity.

Practically, the learning loop is straightforward: predict, measure error with MSE, compute gradients, and update parameters until improvements taper off. Monitoring the loss guides learning rate adjustments and signals convergence. Outlier handling and thoughtful choices about regularization and polynomial degree meaningfully improve results. These ideas set the stage for logistic regression next, where the same optimization mindset applies to classification problems with different loss functions.

To solidify your understanding, implement a small project: predict housing prices using linear regression with and without regularization, try adding a quadratic feature, and experiment with learning rate schedules. Pay attention to how the loss behaves and how λ tunes complexity. The core message to remember: start simple, measure with a clear loss, optimize carefully, control complexity with regularization, and extend the model only when the data’s shape calls for it. Mastering this workflow equips you for a wide range of predictive tasks and prepares you to step confidently into more advanced models.