Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 17: Alignment - RL 2

This lecture focuses on the reinforcement learning (RL) step of aligning large language models (LLMs) to human preferences, building on the standard RLHF (Reinforcement Learning from Human Feedback) pipeline. The instructor begins by revisiting reward hacking—when a model finds clever but undesirable ways to maximize a reward, like outputting very long responses if the reward correlates with length. The RLHF pipeline is reviewed: pre-train a language model, gather human preference data by comparing outputs, train a reward model from these comparisons, and then use RL to fine-tune the policy (the model’s behavior) toward higher rewards while staying close to the original model. The heart of today’s lecture is policy optimization under practical constraints: enormous action spaces, noisy/sparse reward signals, and the need to avoid drifting too far from the base model’s knowledge. A central stabilizing tool is a KL divergence penalty that discourages large deviations from the pre-trained policy, controlled by a hyperparameter beta.

The lecture then explores Proximal Policy Optimization (PPO), the most commonly used policy gradient algorithm for RLHF. PPO’s clipped objective limits how much action probabilities can change in one update. The advantage function—often estimated as reward minus a value prediction—guides which actions to strengthen or weaken. The instructor walks through PPO’s steps for language models: roll out the current policy to collect data, estimate advantages, compute the PPO loss with clipping and optional KL terms, update the model, and repeat. A concrete example uses a text summarization task where humans prefer one summary over another; PPO gradually shifts the policy to favor the preferred option, but only within controlled update sizes.

While PPO is popular, it presents challenges for very large models: high compute and memory demands, sensitivity to hyperparameters such as the clipping range and KL penalty strength, and stability concerns. Alternative RL approaches are mentioned. TRPO (Trust Region Policy Optimization) enforces a trust region by directly constraining KL divergence, with stronger theoretical guarantees but greater implementation complexity due to second-order optimization using the Fisher information matrix. Actor-critic methods learn a separate value function alongside the policy, often improving sample efficiency but sometimes at the cost of stability.

A newer pathway, Direct Preference Optimization (DPO), avoids training a separate reward model entirely. DPO reframes preference learning as a binary classification task over pairs of outputs given a prompt: the model is trained with cross-entropy loss to prefer the human-chosen output. Remarkably, optimizing this loss can be shown to correspond to RL with a particular reward. DPO’s advantages include a simpler pipeline, increased stability (no advantage estimation), and potential sample efficiency. However, its performance depends heavily on high-quality preference data and careful hyperparameter choices. The instructor also addresses how to handle ties in preference data—either by excluding them, down-weighting their loss, or using a margin so only strong preferences drive updates.

The lecture closes with broader RL topics in the context of LLM alignment. Reward shaping is presented as the art of designing reward functions that guide learning, often by layering simple rewards first and gradually adding more nuanced components, sometimes mixing intrinsic (exploration-promoting) and extrinsic (goal-oriented) signals. The exploration–exploitation trade-off is highlighted: models must try new actions to discover better strategies while also exploiting known good actions to earn reward, with methods like epsilon-greedy and upper confidence bounds helping manage the balance. Finally, evaluation metrics must reflect alignment with human values, not just generic language quality. Human evaluation, pairwise comparison accuracy, and task-specific measures are emphasized as better indicators than perplexity alone.

By the end, learners understand the practical algorithms used to align LLMs via RL, especially PPO and DPO, how KL penalties stabilize training, what challenges arise at scale, and how to think about reward shaping, exploration, and evaluation. The audience should come away with a clear blueprint for the RL optimization step in RLHF and a sense of when to prefer PPO versus DPO or consider TRPO/actor-critic variants.

This material suits intermediate practitioners who know basic ML, neural network training, and the idea of language modeling. It is helpful to understand gradients, loss functions, and probability distributions. After this lecture, you will be able to describe and implement the PPO loop for LLMs, explain DPO and its advantages, design KL-constrained objectives, and choose appropriate evaluation metrics for alignment.