Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 9 - Recap & Current Trends

This lecture introduces deep learning, explains why it matters, and contrasts it with traditional ways of building intelligent systems. The instructor starts with a concrete imaging example: given a drone photo that includes a soccer field and buildings, the old approach required hand-written code to detect corners, lines, and surfaces, and then more code to group these into recognizable shapes. That approach works only when new images look similar to those used while designing the rules. As soon as the angle, lighting, time of day, or season changes, those brittle rules often fail. Deep learning changes this by learning directly from data instead of relying on carefully crafted rules.

The lecture positions deep learning as a subset of machine learning. In traditional machine learning, engineers use hand-designed feature extractors to convert raw inputs (like images or text) into numbers. A separate classifier then maps these numbers to outputs (like cat or dog, or English vs. French). Deep learning replaces the hand-designed feature extractors with a neural network that learns features automatically. This is a key shift because it removes a major bottleneck: inventing and maintaining fragile, hand-tuned features.

The instructor introduces neural networks as layered structures made of simple computing elements called neurons. Each neuron computes a weighted sum of its inputs and passes it through a nonlinear activation function. By stacking layers, networks learn features of increasing complexity: early layers detect simple edges, middle layers capture shapes, and later layers recognize whole objects. This bottom-up building of representations makes the system flexible and powerful. The term “deep” refers to the presence of many layers.

The lecture also addresses why deep learning took off only recently, given that neural networks have existed for decades. Three trends came together: (1) far more data became available to train on, (2) much more computing power made training large models possible, and (3) new algorithms and training ideas made optimization of deep networks practical and stable. The synergy of data, compute, and ideas pushed performance past previous ceilings across tasks like vision and speech.

Training neural networks involves showing many examples and telling the network the correct answers (labels). The network adjusts its internal weights to reduce the difference between its predictions and the correct outputs. This adjustment uses backpropagation, an algorithm that computes how each weight contributed to the error and how it should change. With enough data, compute, and good training procedures, networks generalize and perform well on new, unseen inputs.

The lecture briefly mentions different architectures suited to different data types. Fully connected neural networks connect every neuron in one layer to every neuron in the next. Convolutional neural networks (CNNs) are particularly effective for images because they capture local patterns like edges and textures and reuse features across the image. Recurrent neural networks (RNNs) handle sequences such as text and speech by processing inputs step by step and carrying information over time. These architectural choices let deep learning fit many problem domains.

Advantages of deep learning include automatic feature learning, high accuracy, and strong robustness to variations in input conditions. Instead of writing brittle detection rules, we let the model learn features that work across many settings. As a result, deep learning systems often match or surpass human-level performance in recognition tasks. On the flip side, deep learning requires large amounts of labeled data, which can be costly to collect. Another drawback is interpretability: these systems often act like black boxes, making it hard to explain individual decisions—an issue for sensitive applications like medical diagnosis.

By the end of the lecture, you understand what deep learning is, how it differs from traditional machine learning, the basic structure and training of neural networks, and the reasons for its recent success. You also learn the main pros and cons, along with examples across images and text. This sets the stage for later lectures that dive into applications and deeper technical details of architectures, training methods, and deployment.

Overall Architecture/Structure

1. Problem framing and inputs

• We start with a task such as recognizing objects in a drone image or identifying the language of a sentence. The raw inputs can be pixels (for images) or tokens/characters (for text). The traditional pipeline separated feature engineering from classification. Deep learning merges these into one learned model.

1. Neural network as the core engine

• A neural network is built from layers. Each layer contains neurons that compute y = activation(w·x + b), where w are weights, x are inputs from the previous layer, and b is a bias term. The activation is a nonlinear function that lets the network learn complex patterns beyond simple linear rules. Stacking multiple layers builds a hierarchy of features.

1. Feature hierarchy and representation learning

• The network learns to transform raw inputs into more useful internal representations. In images: edges and corners appear early; shapes and textures appear mid-layer; whole objects emerge late. In text: early processing may detect character/word patterns; later layers capture phrases or sentence-level meaning. This progressive abstraction is learned from data rather than hand-designed.

1. Output layer and decision

• The final layer produces task-specific outputs. For classification (cat vs. dog; English vs. French), it outputs scores or probabilities for each class. The model’s prediction is compared to the correct label to measure error. This error guides the weight updates during training.

1. Training loop

• Training cycles through many examples. For each input, the network performs a forward pass (computes outputs) and then a backward pass (computes gradients of the error with respect to each weight). Using these gradients, it updates weights slightly (often using gradient-based optimizers) to reduce future error. Repeating this across the dataset teaches the model to generalize.

Roles of Components

• Inputs: raw data such as pixel arrays or sequences of tokens.
• Weights and biases: learnable parameters that define what each neuron computes.
• Activation functions: nonlinearities like ReLU or sigmoid that let networks model complex relationships (the lecture mentions nonlinearity abstractly without naming types).
• Layers: collections of neurons that compute transformations; depth refers to the count of these layers.
• Architectures: fully connected layers for general transformations; CNNs for spatial images; RNNs for sequences.
• Loss/error: measures mismatch between prediction and truth; drives learning.
• Backpropagation: computes gradients to update weights; makes training deep networks feasible.

Data Flow

• Forward pass: Input → Layer 1 transform → Layer 2 transform → … → Output predictions.
• Loss computation: Compare predictions to ground-truth labels to get an error signal.
• Backward pass: Propagate error signal backward to compute gradients for each weight.
• Weight update: Adjust weights by small steps in the direction that reduces the loss.
• Iterate: Repeat for many batches and epochs until performance stabilizes.

Code/Implementation Details (Conceptual, no code shown in lecture)

• Language/framework: While the lecture does not show code, common tools include Python with PyTorch or TensorFlow. The ideas map directly to these frameworks: you define layers, specify a loss, and run training loops.
• Layers: Fully connected (linear/dense) layers implement matrix multiplies plus bias, followed by activation. CNN layers implement convolution operations that slide filters over the input. RNN layers implement step-by-step processing with hidden state carried forward.
• Parameters: Weights are usually initialized randomly. Biases are small constants. During training, parameters change to fit data.
• Activations: Nonlinear functions ensure the model can represent complex functions; without them, stacked linear layers would collapse into a single linear transform.
• Loss functions: For classification, losses measure how wrong class probabilities are compared to true labels. The lecture references error in general terms; conceptually, the loss quantifies this error.
• Optimizers: Gradient descent and its variants update weights using gradients from backprop. The lecture names backprop as the core gradient computation method.
• Training data: Pairs of inputs and labels drive supervised learning. The model learns to map inputs to outputs by minimizing loss.

Tools/Libraries Used (Conceptual)

• Deep learning libraries (e.g., PyTorch, TensorFlow) abstract layers, activations, losses, and optimizers. They automate backpropagation via automatic differentiation. Dataloaders help feed batches of examples. GPU support speeds up matrix operations.
• While not specified in the lecture, GPUs are a major reason compute scaled; they accelerate the heavy linear algebra behind deep learning.

Step-by-Step Implementation Guide (Conceptual Walkthrough)

• Step 1: Define the task and gather data. For image recognition, collect many labeled images across varied conditions (angles, lighting, seasons). For language ID, collect sentences labeled by language (English, French, etc.). Ensure diversity so the model learns robust features.
• Step 2: Choose an architecture. Start with a simple fully connected network for small tabular inputs. Use a CNN for images to exploit spatial structure. Use an RNN (or sequence model) for text or time series to handle order and context.
• Step 3: Initialize the model components. Specify layers (input size, hidden sizes, output size). Choose activations (any standard nonlinearities suffice conceptually). Set up a loss function appropriate for classification.
• Step 4: Prepare the training loop. For each batch: run a forward pass to get predictions; compute loss against labels; run backpropagation to compute gradients; update weights with a chosen optimizer. Repeat across many epochs until validation performance improves and stabilizes.
• Step 5: Evaluate on new data. Test on images or sentences not seen during training. Check robustness across varied conditions (different angles, lighting, times). If performance drops, add more data or adjust architecture/training.
• Step 6: Iterate and improve. Increase data variety, tune model depth, and adjust learning rate or batch size (general training knobs) to enhance learning. Reassess architecture choices (e.g., try a CNN for vision if a fully connected network struggles).

Tips and Warnings

• Data quantity matters: Deep learning thrives on large datasets. If data is scarce, performance may suffer. Collect more diverse examples to improve robustness.
• Match architecture to data: Use CNNs for images and RNNs for sequences. Fully connected layers are general but may be inefficient for high-dimensional spatial inputs.
• Beware of the black box: Deep models can be hard to interpret. In sensitive applications, plan for methods that provide explanations or confidence measures.
• Training stability: Ensure inputs and labels are correctly paired and preprocessed consistently. Monitor training and validation performance to detect issues early.
• Compute needs: Training deep models requires significant compute. Use appropriate hardware to keep training times reasonable.

Connecting Back to the Lecture’s Examples

• Drone imagery: A hand-coded pipeline uses corner/line/surface detectors to trace buildings and fields. Deep learning instead learns features directly from many diverse drone images, making it robust to changes in angle and lighting.
• Image classification: Instead of designing edge histograms or texture descriptors, the network learns from labeled images of cats, dogs, cars, or soccer fields. Early layers detect edges; deeper layers identify objects.
• Language tasks: For detecting English vs. French or translating a sentence, a sequence model learns patterns in character or word order. The model uses many examples to map inputs to outputs, updated via backprop.

Why Depth, Data, Compute, and Ideas Align

• Depth makes hierarchical representation learning possible, enabling recognition of complex objects. Data diversity teaches invariance to conditions like lighting and angle. Compute power trains large networks fast enough to be practical. New training ideas improved optimization and stability. Together, they transformed performance and usability.

Advantages and Disadvantages in Practice

• Advantages: Automatic feature learning reduces manual engineering time and improves robustness. High accuracy across varied tasks means strong performance in real-world conditions. Flexibility across data types lets one learning framework handle images, text, and sequences.
• Disadvantages: Requires lots of labeled data, which can be expensive. Complexity makes decisions hard to explain, the black box problem. These trade-offs must be weighed, especially in high-stakes settings.

Summary

• Deep learning replaces hand-crafted features with learned representations built by neural networks. Layers learn increasingly complex features, enabling robust recognition. Backpropagation and modern compute allow training at scale. The approach’s strengths and weaknesses reflect its dependence on data and its complexity. This foundation supports a wide range of applications that the rest of the course will explore in depth.

This lecture defined deep learning as learning directly from data using multi-layer neural networks and contrasted it with traditional rule-based and classic machine learning approaches. Instead of handcrafting features like corners and lines, deep learning models learn features automatically and build hierarchical representations—edges, shapes, then full objects. The training process uses many labeled examples and backpropagation to adjust weights and reduce prediction error. Three forces—abundant data, powerful compute, and better training ideas—sparked the deep learning revolution and made large, accurate models practical. Different architectures fit different data types: fully connected networks for general transformations, CNNs for images, and RNNs for sequences. The main strengths are high accuracy and robustness; the main trade-offs are large data needs and the black box interpretability problem.

For practice, try building a simple classifier on a small image dataset, first with handcrafted features and then with a small neural network, to feel the difference. Experiment with changes in lighting and angle to see which approach holds up. Next, build a tiny language ID model using character-level inputs to observe how sequence handling helps. Keep notes on how data diversity improves results.

As next steps, dive deeper into architectures like CNNs for vision and RNNs for sequences, and later explore modern variants. Study training techniques and evaluation methods, and learn about ways to interpret models in sensitive domains. The core message to remember is this: deep learning learns features and decisions together, trading manual rules for data-driven representation learning. With the right data and compute, it can deliver robust, accurate systems that adapt to the real world.