Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 4 - LLM Training

This lecture teaches you how to effectively train artificial neural networks by walking through core optimization algorithms, learning rate strategies, regularization methods, and stability techniques used in practice. It begins with a short recap of gradient descent (GD) and stochastic gradient descent (SGD), which update model parameters by moving in the opposite direction of the gradient of a loss function. You will see why mini-batches are the practical way to compute gradients and how the noise they introduce can help a model escape poor local minima. Then it moves into more advanced optimizers—Momentum, RMSprop, and Adam—explaining how each one improves training speed and stability by smoothing or adapting learning steps.

Next, the lecture covers learning rate decay schedules. These schedules start with a relatively large learning rate so training can make fast progress early on, and then reduce the learning rate over time to refine the solution. You’ll learn three common methods: step decay (reduce by a fixed factor every few epochs), exponential decay (shrink continuously by a constant rate per epoch), and cosine annealing (smoothly decreasing the rate following a cosine curve from a maximum to a minimum). Each method provides a different shape of decrease, and the idea is to achieve both quick initial learning and careful fine-tuning near the end.

Regularization is another big topic in this lecture. Regularization is crucial when a model might memorize the training data and not generalize well to new data, a problem known as overfitting. You’ll learn L2 regularization (also called weight decay), which encourages small weights by adding a penalty on the sum of squared weights, and L1 regularization, which encourages sparsity by penalizing the sum of absolute values of weights. The lecture also explains Dropout, a method where you randomly set some neurons’ outputs to zero during training. Dropout acts as a kind of ensemble, preventing any single neuron from dominating and improving generalization; at test time, no neurons are dropped.

To improve training stability and speed, the lecture introduces batch normalization. Batch normalization normalizes the activations of each layer using the mean and variance computed over the current mini-batch. After normalization, it applies a learned scale (gamma) and shift (beta), typically starting at gamma=1 and beta=0. This process allows the model to stabilize intermediate values in the network, often enabling larger learning rates and faster convergence.

The lecture closes with practical, experience-based tips. Choose an architecture appropriate to the task and the amount of data: deeper and wider networks are more powerful but require more data and regularization. Preprocess inputs by normalizing or scaling features and removing obvious outliers because neural networks are sensitive to feature scales. Initialize weights properly—randomly and with methods like Xavier or He initialization—to maintain healthy signal flow. Pick a reasonable learning rate and consider adding a decay schedule. Monitor training and validation losses to detect overfitting, visualize weights to catch issues early, and always maintain a representative validation set for tuning hyperparameters like learning rate and regularization strength.

Who is this lecture for? It is aimed at beginners to intermediate learners who have basic familiarity with machine learning and neural networks (e.g., knowing what a loss function is and what a model’s parameters are). You do not need deep math to benefit, but comfort with gradients and the idea of optimization helps. After completing the lecture, you will be able to pick and configure optimizers like Momentum, RMSprop, and Adam; design and apply learning rate decay schedules; use regularization techniques such as L1, L2, and Dropout; apply batch normalization; and follow a practical training workflow with data preprocessing, weight initialization, monitoring, and validation.

The lecture is structured as follows: it starts with a recap of basic gradient-based training and mini-batch SGD. It then introduces three optimizers—Momentum, RMSprop, and Adam—explaining their update rules and intuitions. Next come learning rate decay strategies, including step, exponential, and cosine annealing, with their formulas and roles. After that, it dives into regularization: L2, L1, and Dropout, with the effects each has on weights and generalization. Then it explains batch normalization and why it improves training stability and performance, including typical initialization of its parameters. Finally, it wraps up with a checklist of practical training tips: architecture selection, preprocessing, initialization, learning rate choice, monitoring, weight visualization, and using a proper validation set.

Overall Architecture/Structure of Training

1. Objective and Data Flow

• We define a loss function L(theta) that measures how far the network’s predictions are from targets. Training data is processed in mini-batches. For a mini-batch B, we perform a forward pass to compute outputs and the batch loss L_B(theta). Then, we run backpropagation to compute the gradient of the loss with respect to parameters, grad L_B(theta). Finally, we update parameters according to an optimization rule. This loop repeats over many epochs (full passes through the training dataset).

1. Gradient Descent and Stochastic Gradient Descent

• Gradient Descent (GD): theta := theta − eta * grad L(theta), where eta is the learning rate. This is exact only if we compute the gradient on the whole dataset each step (which is slow). Stochastic Gradient Descent (SGD) approximates this by using a mini-batch. For a mini-batch B, we compute grad L_B(theta), and update theta := theta − eta * grad L_B(theta). The mini-batch size controls the trade-off between noisy gradients (small batches) and better gradient estimates (large batches). Noise in SGD can help escape local minima or saddle points because it prevents the optimizer from getting stuck in flat or shallow areas.

1. Momentum Optimizer

• Motivation: Vanilla SGD can zig-zag across narrow valleys where gradients alternate signs across dimensions. Momentum adds memory of past updates to smooth the trajectory.
• State and Updates: Maintain velocity v, initialized to zero. Each step: v := mu * v − eta * grad L_B(theta). Then theta := theta + v. The hyperparameter mu (momentum coefficient) is often around 0.9. Intuition: If grads mostly point the same way, v grows in magnitude, taking larger steps. If grads reverse, the previous v opposes the change, damping oscillations. Momentum effectively acts like an exponentially weighted average of past gradients scaled by eta.

1. RMSprop Optimizer

• Motivation: Different parameters may have gradients of very different magnitudes. A single global learning rate can be too big for some and too small for others.
• State and Updates: Maintain S (same shape as theta) as an exponential moving average of squared gradients: S := beta * S + (1 − beta) * (grad^2) (element-wise square). Update parameters using a normalized gradient: theta := theta − eta * grad / sqrt(S + epsilon). Here, epsilon is a small constant (e.g., 1e−8) added for numerical stability to avoid division by zero. Intuition: If a parameter’s gradient is consistently large, S grows and reduces its effective step size; if small, S shrinks and increases the step.

1. Adam Optimizer

• Motivation: Combine momentum’s first-moment averaging and RMSprop’s second-moment scaling to get robust, adaptive updates with minimal tuning.
• State: Keep M (first moment of gradients) and V (second moment of gradients) for each parameter. Initialization usually M=0, V=0. At step t: M := beta1 * M + (1 − beta1) * grad. V := beta2 * V + (1 − beta2) * (grad^2). Bias correction: M_hat := M / (1 − beta1^t), V_hat := V / (1 − beta2^t). Update: theta := theta − eta * M_hat / (sqrt(V_hat) + epsilon). Defaults: beta1=0.9, beta2=0.999, epsilon=1e−8. Intuition: M captures direction (smoothed gradient), V adjusts step size per parameter (like local curvature sensitivity), and bias correction fixes the early-step underestimation caused by initializing M and V at zero.

1. Learning Rate Decay Schedules

• Why decay? Early in training, larger steps help explore and move quickly toward a good area; later, smaller steps allow precise adjustment near a minimum.
• Step Decay: eta := eta * gamma at specified epochs (e.g., every 30 epochs), with gamma < 1 (like 0.1). Pros: simple, known strong baselines. Cons: sudden changes may destabilize briefly.
• Exponential Decay: eta_t = eta_0 * k^t with 0 < k < 1. Pros: smooth and continuous. Cons: need to pick k carefully so it neither decays too fast nor too slow.
• Cosine Annealing: eta(t) = eta_min + 0.5 * (eta_max − eta_min) * (1 + cos(pi * t / T_max)), for t in [0, T_max]. Pros: smooth start and smooth finish; often yields better final performance. Cons: need to choose eta_min, eta_max, and T_max thoughtfully; typical to pick eta_min near zero and eta_max as your initial learning rate.

1. Regularization Methods

• Purpose: Limit model complexity and improve generalization by discouraging overfitting to training data.
• L2 Regularization (Weight Decay): Modify the loss to L_total = L(theta) + (lambda/2) * ||theta||^2, where ||theta||^2 is the sum of squared weights. The gradient of the L2 term is lambda * theta, so each update shrinks weights slightly, pulling them toward zero. This reduces variance in predictions and makes the model more robust to noise.
• L1 Regularization: Modify the loss to L_total = L(theta) + lambda * ||theta||_1, where ||theta||_1 is the sum of absolute weight values. The subgradient for a weight w is lambda * sign(w) (or a value in [−lambda, lambda] when w=0). This tends to push many weights exactly to zero, producing sparse models that can be easier to interpret and more efficient.
• Dropout: During training, independently set each neuron’s output to zero with probability p (e.g., p=0.5 for a hidden layer). This prevents neurons from co-adapting and forces the network to distribute learned representations. At test time, no units are dropped. In code, you either scale activations during training (inverted dropout) or at test time so that expected activations match.

1. Batch Normalization

• Purpose: Stabilize and speed training by normalizing intermediate activations.
• Per mini-batch and per activation channel (or per neuron), compute mean μ_B and variance σ_B^2. Normalize: x_hat = (x − μ_B) / sqrt(σ_B^2 + epsilon). Then apply learned scale and shift: y = gamma * x_hat + beta. Initialize gamma=1 and beta=0 so the layer initially just normalizes. Over training, gamma and beta let the network choose the optimal scale and offset. Benefits include more stable gradients, the ability to use higher learning rates, and often improved generalization.

1. Practical Training Tips

• Architecture Selection: Match model capacity (layers and neurons) to data size and task complexity. Too small: underfitting; too large: overfitting without sufficient regularization and data.
• Data Preprocessing: Normalize or scale features to similar ranges (e.g., zero mean, unit variance). Remove or limit the influence of obvious outliers that can skew gradients. Neural nets are sensitive to feature scale; consistent scales make optimization more stable and efficient.
• Weight Initialization: Avoid all-zeros initialization because it makes all neurons compute the same thing and gradients identical, preventing learning. Random uniform/normal breaks symmetry. Xavier (Glorot) initialization scales weights based on fan-in and fan-out to keep activation variance steady, commonly for tanh/sigmoid. He initialization scales weights for ReLU-like activations to keep forward and backward variances healthy.
• Learning Rate Selection: Start with a reasonable base rate (e.g., 1e−3 for Adam, task-dependent for SGD) and consider a decay schedule. Too high causes divergence or chaotic loss; too low slows learning excessively.
• Monitoring: Track both training and validation losses. If training improves while validation worsens, apply stronger regularization, reduce capacity, or gather more data. Visualize weights and sometimes activations; extremely large or tiny weights suggest adjusting learning rate or regularization.
• Validation Set: Reserve a representative validation set to tune hyperparameters (learning rate, regularization strength, dropout rate, etc.). This set should reflect the data your model will see in the real world.

Implementation Details and Execution Flow (Pseudo-Workflow)

1. Initialize: Choose architecture (layers, neurons), activation functions, and loss. Initialize weights (e.g., He for ReLU), biases to small values (often zero). Set optimizer (e.g., Adam with defaults). Pick an initial learning rate and a learning rate decay schedule.
2. Preprocess Data: Normalize or standardize inputs; optionally clip or handle outliers. Split data into training and validation sets.
3. Training Loop per Epoch: a) Shuffle training data and form mini-batches (e.g., batch size 32–256). b) For each mini-batch B: do forward pass to compute predictions and loss L_B. c) Run backprop to compute gradients of L_B with respect to all parameters. d) Update optimizer state (e.g., M, V in Adam; S in RMSprop; v in Momentum) and then update parameters. e) Optionally, apply batch normalization and dropout during forward passes as configured (dropout only during training).
4. End of Epoch: Compute training loss/accuracy and validation loss/accuracy. Adjust learning rate if using step decay at scheduled epochs; otherwise, the scheduler updates it automatically. Check for overfitting signs and adjust regularization or capacity if needed.
5. Repeat: Continue for T_max epochs or until convergence criteria (e.g., validation stops improving). Save the best model based on validation metrics.

Tools and Libraries (Conceptual)

• Although not shown in code here, in practice you would use a deep learning library like PyTorch or TensorFlow. These frameworks provide automatic differentiation (backprop), built-in optimizers (SGD, Momentum, RMSprop, Adam), learning rate schedulers (step, exponential, cosine), regularization utilities (weight decay parameter in optimizers), dropout and batch norm layers, and monitoring tools (TensorBoard or custom logging).

Tips and Warnings

• Learning Rate Pitfalls: If loss explodes or becomes NaN, reduce learning rate, check for too-small epsilon in RMS/Adam, or reduce batch size. Try gradient clipping if gradients explode.
• Regularization Balance: Too much regularization leads to underfitting; too little causes overfitting. Tune lambda (L1/L2) and dropout p on the validation set.
• Batch Size Interactions: Larger batches yield smoother gradients but may generalize slightly worse; smaller batches add noise which can help escape sharp minima. Batch norm’s behavior depends on batch size; too tiny batches may give noisy statistics.
• Initialization Mismatches: Using Xavier with ReLU may cause vanishing activations; prefer He for ReLU families. Check early-layer activation histograms to verify healthy signal propagation.
• Schedule Sensitivity: Step decay requires choosing the right epochs for drops; exponential decay needs a sensible k; cosine annealing needs well-chosen eta_max, eta_min, and T_max. If convergence stalls late, lower the floor (eta_min) or extend T_max.
• Validation Leakage: Never tune on the test set; only use validation for hyperparameters. Keep the validation set distribution close to the real deployment setting.

Putting It All Together

• A solid baseline recipe: preprocess inputs (standardize), initialize with He (for ReLU networks), use Adam (beta1=0.9, beta2=0.999, epsilon=1e−8) with an initial learning rate around 1e−3, and apply a cosine annealing schedule over your planned epochs. Add L2 weight decay and dropout if the model is large or overfits. Include batch normalization to stabilize training and allow a higher starting learning rate. Monitor training/validation curves, adjust regularization and learning rate if validation worsens, and always select final hyperparameters based on validation performance.

This end-to-end approach mirrors what practitioners do daily and reflects the lecture’s central message: combine the right optimizer, learning rate strategy, regularization, and normalization with careful monitoring to train neural networks that not only fit the training data but also generalize well.

This lecture provided a practical roadmap for training neural networks effectively and robustly. It began with the core idea of minimizing a loss function using gradients, comparing full gradient descent with mini-batch SGD. It then introduced momentum to speed learning along stable directions, RMSprop to adapt learning rates per parameter based on squared gradients, and Adam to combine the strengths of both with bias correction—often a strong default optimizer. Learning rate schedules—step, exponential, and cosine annealing—were explained as tools to start training fast and finish carefully, with cosine annealing offering a particularly smooth and effective curve.

To combat overfitting, the lecture highlighted regularization methods. L2 (weight decay) keeps weights small and stable, L1 encourages sparsity by zeroing out many weights, and Dropout randomly disables neurons during training to reduce co-adaptation and improve generalization. Batch normalization further stabilizes and speeds training by normalizing activations per mini-batch and then learning the best scale and shift, with gamma initialized to 1 and beta to 0.

Finally, a set of practical tips tied everything together: choose an architecture that matches your data and task complexity; preprocess inputs through normalization and outlier handling; initialize weights with methods like Xavier or He to maintain healthy activation variance; pick a reasonable learning rate and pair it with a decay schedule; and continuously monitor both training and validation losses. Visualize weights to spot extremes and use a representative validation set for hyperparameter tuning, ensuring that your model’s improvements hold up on data it hasn’t seen.

The overarching message is that successful training is a combination of the right optimizer, a thoughtful learning rate strategy, appropriate regularization, and stability techniques like batch normalization, all guided by careful monitoring on a validation set. By following these principles, you can train networks that learn quickly, avoid common pitfalls, and most importantly, generalize well to new data.