Stanford CS230 | Autumn 2025 | Lecture 1: Introduction to Deep Learning

This opening lecture of Stanford CS230, taught by Andrew Ng, introduces deep learning, sets the course roadmap, explains logistics and expectations, and begins the technical foundations with perceptrons, activation functions, gradient descent, multi-layer perceptrons (MLPs), and backpropagation. It starts by defining deep learning as a subset of machine learning that uses neural networks with many layers to learn complex, hierarchical representations from data. A practical intuition is given through images: early layers detect simple edges, later layers detect textures and parts, and deeper layers recognize whole objects. The lecture emphasizes why deep learning has become so powerful: access to large datasets, flexible architectures that can be adapted to many tasks, and the capacity to model non-linear relationships.

The course is structured in three parts. Part 1 builds the foundation: neural networks, backpropagation (the algorithm to compute gradients efficiently), optimization algorithms (like gradient descent variants), and regularization techniques (methods to reduce overfitting). Part 2 focuses on convolutional neural networks (CNNs) for visual tasks including image recognition, object detection, and segmentation. Part 3 introduces recurrent neural networks (RNNs) to handle sequential data such as natural language, speech, and time series. Throughout, the course balances theory and hands-on practice, using Python and TensorFlow as the main tools. Students are encouraged to brush up on these technologies with provided tutorials and external resources.

Logistics are set with clear expectations: the class is demanding and requires consistent effort. There are four programming homeworks providing hands-on experience, a midterm exam testing theoretical understanding, and a final project where students apply deep learning to a real-world problem. Grading weights are 40% for homeworks, 20% for the midterm, and 40% for the final project. Support includes section leading meetings/office hours for one-on-one help, plus an online Q&A forum for community-driven assistance.

The technical portion of the lecture begins with the single-layer perceptron, the simplest neural network building block. Each neuron computes a weighted sum of inputs plus a bias, passes it through a non-linear activation function, and produces an activation (output). Common activation functions include sigmoid (outputs between 0 and 1), ReLU (outputs 0 for negative inputs and the input itself for positive inputs), and tanh (outputs between −1 and 1). The lecture explains why activation functions are essential: they introduce non-linearity, allowing the model to learn beyond straight-line relationships. Training a perceptron involves finding weights and bias that minimize prediction error, using gradient descent to iteratively update parameters in the direction that most reduces the error.

The lecture then generalizes to multi-layer perceptrons (MLPs), which stack multiple layers: input, one or more hidden layers, and an output layer. Each layer transforms the previous layer’s activations using its own weights, bias, and activation function. The crucial difference from a single-layer perceptron is that hidden layers allow the network to represent highly non-linear decision boundaries and hierarchical abstractions of data. Finally, the backpropagation algorithm is introduced as the key method to compute gradients for all weights and biases efficiently by doing a forward pass (compute activations) and a backward pass (propagate errors and compute derivatives).

By the end of this lecture, you know what deep learning is, where it’s used (image recognition, NLP, speech recognition), why it has become impactful, and how this course will build your knowledge step-by-step. You also gain the foundational equations of neurons and the training loop (forward computation and gradient-based updates), preparing you for deeper dives into optimization, regularization, CNNs, and RNNs in the coming sessions. The lecture’s core message is that deep learning’s strength lies in learning layered, non-linear representations from data, and that with disciplined study and practice, you can master these tools to solve meaningful real-world problems.

Overall Architecture/Structure of Concepts Introduced:

1. Single-Layer Perceptron (SLP)

• Role: The perceptron is the basic computational unit of neural networks. It maps an input vector x ∈ R^d to an output a via two steps: an affine transform z = w^T x + b, then a non-linear activation a = g(z).
• Data Flow: Inputs (features) enter as x. Parameters include the weight vector w and bias b. The neuron computes z, then applies an activation function g to produce a. In classification, a may be a probability-like score (e.g., with sigmoid) or a transformed feature for deeper layers.
• Why Non-Linearity: If g were the identity (no activation), the model would be linear: any composition of linear transforms is still linear. Real tasks need non-linear decision surfaces; hence g must be non-linear.

1. Activation Functions

• Sigmoid: g(z) = 1 / (1 + e^(−z)), output in (0, 1). Useful when outputs are interpreted as probabilities; has smooth gradients but can saturate for large |z|.
• ReLU: g(z) = max(0, z). Efficient, reduces vanishing gradients for positive z, sparse activation; can cause “dead” neurons if inputs stay negative.
• Tanh: g(z) = (e^z − e^(−z)) / (e^z + e^(−z)), output in (−1, 1). Zero-centered, but can still saturate for large |z|. Choice of activation affects training dynamics, gradient flow, and performance.

1. Loss/Error and Gradient Descent Training

• Objective: Minimize a loss E(w, b) that measures discrepancy between predictions and labels. The exact loss wasn’t specified here, but common ones include mean squared error (MSE) for regression and cross-entropy for classification.
• Gradient Descent Update: Repeatedly update parameters by moving opposite to the gradient: w := w − α ∂E/∂w; b := b − α ∂E/∂b. α is the learning rate; it scales the step size.
• Iteration Loop: For each batch or dataset pass, compute predictions (forward pass), compute loss, compute gradients (backward pass or analytic derivatives for SLP), then update parameters.

1. Multi-Layer Perceptron (MLP)

• Layers: Input layer a^(0) = x; for l = 1..L: z^(l) = W^(l) a^(l−1) + b^(l), a^(l) = g(z^(l)). W^(l) is a matrix mapping from layer l−1 to l; b^(l) is the bias vector.
• Expressiveness: With enough hidden units/layers and appropriate activations, MLPs can approximate very complex functions (universal approximation). Hidden layers create non-linear feature transformations.
• Output Layer: May use sigmoid for binary classification, softmax (though not spelled out here) for multi-class, or linear for regression.

1. Backpropagation (High-Level)

• Purpose: Efficiently compute gradients ∂E/∂W^(l) and ∂E/∂b^(l) for all layers.
• Mechanism: Store activations and pre-activations during forward pass. During backward pass, compute output layer error signal (e.g., δ^(L) = ∂E/∂z^(L)), then recursively propagate to earlier layers via δ^(l) = (W^(l+1))^T δ^(l+1) ⊙ g′(z^(l)), where ⊙ is elementwise multiplication. Then gradients are ∂E/∂W^(l) = δ^(l) (a^(l−1))^T and ∂E/∂b^(l) = δ^(l).
• Efficiency: Avoids redundant computations by reusing stored forward-pass values. Complexity scales linearly with number of parameters.

Detailed Explanation of Each Component: A) Inputs, Weights, and Bias

• Inputs x: A vector of features representing the sample. For images, this could be pixel intensities; for text, it could be counts or embeddings; for audio, spectral features.
• Weights w (or W): Learnable parameters that scale and combine inputs. Intuitively, each weight indicates how much its corresponding input contributes to the neuron’s decision.
• Bias b: A learnable offset allowing the activation function’s threshold to shift independently of the inputs, enabling the model to fit patterns not passing through the origin.

B) Activation Function Behavior and Choice

• Sigmoid Pros/Cons: Pros include probabilistic interpretation and smoothness; cons include saturation (gradients approach zero for large |z|) and lack of zero-centering which can slow optimization.
• ReLU Pros/Cons: Pros are computational simplicity, reduced vanishing gradient for z > 0, and sparse activations that can regularize implicitly. Cons include dead neurons when weights/biases force z < 0 persistently, and unbounded positive outputs.
• Tanh Pros/Cons: Zero-centered outputs can aid optimization; still suffers from saturation in extremes. Often used historically, now frequently replaced by ReLU-family activations in deep networks.
• Importance: The activation directly shapes the gradient signal g′(z), affecting training speed and stability.

C) Gradient Descent Mechanics

• Gradient Interpretation: ∂E/∂w_i tells how much the loss would change if weight w_i changed slightly. Moving in the negative gradient direction reduces the loss most rapidly (locally).
• Learning Rate Tuning: α too large causes oscillations or divergence; α too small slows training. Practical strategies (not covered here) include decay schedules or adaptive methods, but the core idea is to pick α that steadily decreases loss.
• Stopping Criteria: Iterate until convergence (loss stabilizes), a set number of epochs, or validation performance stops improving.

D) Forward Pass in MLPs

• Step-by-Step: Given input a^(0) = x, compute z^(1), a^(1), then z^(2), a^(2), and so on until z^(L), a^(L). The final a^(L) is the prediction.
• Data Shapes: If a^(l−1) has n_(l−1) units and a^(l) has n_l units, then W^(l) is n_l × n_(l−1), b^(l) is n_l × 1, a^(l−1) is n_(l−1) × 1, and a^(l) is n_l × 1.
• Vectorization: Implementations compute many samples in parallel by stacking inputs into matrices (batches). This uses efficient linear algebra operations.

E) Backward Pass in MLPs (Conceptual)

• Output Error: Compute δ^(L) from the derivative of the loss with respect to z^(L). For example, with a sigmoid output and cross-entropy loss, δ^(L) often simplifies to (a^(L) − y).
• Propagation: For l from L−1 down to 1, compute δ^(l) = (W^(l+1))^T δ^(l+1) ⊙ g′(z^(l)). This links the error in each layer to the next, scaled by the derivative of the activation.
• Parameter Gradients: ∂E/∂W^(l) = δ^(l) (a^(l−1))^T, ∂E/∂b^(l) = δ^(l). Update parameters using these gradients with learning rate α.

F) Importance of Data and Flexibility

• Data Scale: Deep learning performance typically improves with more high-quality data, as models can learn richer patterns and reduce overfitting risk via diversity.
• Flexibility: The same basic framework (layers + activations + gradient-based training) adapts to images (with CNN layers), sequences (with RNN layers), and more.

Tools/Libraries Used (Per Course Setup):

• Python: A general-purpose programming language favored for data science due to readability and rich ecosystem (NumPy, pandas, Matplotlib, etc.).
• TensorFlow: A deep learning framework handling tensor operations, automatic differentiation (autograd), and optimizers. High-level APIs let you define layers (Dense), models (Sequential/Functional), and training loops (fit) without manual gradient coding.
• Why Chosen: Strong community support, performance optimizations, and integration with hardware accelerators (GPUs/TPUs). It streamlines implementing the forward/backward computations discussed.

Step-by-Step Implementation Guide (Conceptual, aligned to lecture content):

1. Define the Model Structure

• For a perceptron: decide input dimension d (number of features), choose an activation (sigmoid/ReLU/tanh), and initialize w ∈ R^d and b ∈ R.
• For an MLP: choose number of layers, units per layer, and activation per layer; initialize W^(l) and b^(l).

1. Forward Pass

• Compute z = w^T x + b (perceptron) or z^(l) = W^(l) a^(l−1) + b^(l) for each layer.
• Apply activations to get a or a^(l) = g(z^(l)). Keep a^(l) and z^(l) for backprop.

1. Compute Loss

• Compare predictions to true labels to get scalar loss E. For binary classification, often use a sigmoid on output and a cross-entropy style loss; for regression, MSE is common.

1. Backward Pass (Backprop)

• Compute gradients of loss w.r.t. outputs, then recursively compute δ^(l) for each layer using chain rule and stored z^(l), a^(l−1).
• Get gradients ∂E/∂W^(l), ∂E/∂b^(l) for all layers.

1. Parameter Update

• Update W^(l) := W^(l) − α ∂E/∂W^(l), and b^(l) := b^(l) − α ∂E/∂b^(l) for each layer. Repeat for multiple iterations/epochs.

1. Evaluation and Iteration

• Monitor training loss and accuracy; check validation performance to ensure generalization. Adjust learning rate, model size, or data preprocessing as needed.

Tips and Warnings (Aligned with Concepts Introduced):

• Activation Choice: Start with ReLU in hidden layers for efficiency; use sigmoid/tanh if your task’s output needs bounded ranges. Beware of saturation with sigmoid/tanh; gradients can vanish when |z| is large.
• Learning Rate: Test a few α values; too large will cause divergence (loss explodes), too small will stall learning. Plot loss over iterations to diagnose convergence behavior.
• Initialization: While not detailed in the lecture, remember that poor initialization can hamper learning; common practice uses small random values to break symmetry.
• Overfitting and Regularization: The course will cover regularization later; for now, note that complex networks can memorize training data—monitor validation metrics.
• Data Preprocessing: Normalize or standardize inputs to help training stability; unscaled features can slow or destabilize gradient descent.

Connecting to Applications (Images, Language, Speech):

• Images: Replace fully-connected layers at the input with convolutional layers to exploit locality and translation invariance; still trained by backprop + gradient descent.
• Language/Speech: Use recurrent (or other sequential) layers to handle time dependencies; training loop remains forward + backward passes with parameter updates.

Putting It All Together—Mental Model:

• A neural network is a chain of affine transforms plus non-linear activations. Training repeatedly performs: forward pass → loss → backward pass → parameter update. Depth and non-linearity let the model learn hierarchical features. Enough data and thoughtful training make it powerful on complex tasks like vision, NLP, and speech.

This first CS230 lecture sets the tone, scope, and foundation for mastering deep learning. You learned that deep learning uses multi-layer neural networks to learn hierarchical, non-linear representations, which is why they work so well for complex tasks like image recognition, natural language processing, and speech recognition. The technical core introduced the perceptron (z = w^T x + b, a = g(z)), emphasized the necessity of non-linear activation functions (sigmoid, ReLU, tanh), and explained gradient descent as the engine that updates weights and bias to minimize error. Extending to multi-layer perceptrons, you saw how stacking layers transforms simple operations into powerful function approximators. Backpropagation ties everything together by efficiently computing gradients layer by layer through a forward and backward pass.

On the course roadmap, expect a balanced blend of theory and practice: Part 1 (foundations: neural networks, backpropagation, optimization, regularization), Part 2 (CNNs for images, object detection, segmentation), and Part 3 (RNNs for sequences: language, speech, time series). Plan for a steady workload with four coding homeworks, a theory-focused midterm, and a real-world final project, graded 40% homeworks, 20% midterm, and 40% final project. You’ll use Python and TensorFlow and get support through office hours and an online Q&A forum.

To practice right away, review the neuron equations, experiment with simple activations on toy inputs, and implement a minimal perceptron or tiny MLP. Try different learning rates to see how convergence changes, and get comfortable inspecting losses and predictions. As next steps, be ready to dive deeper into backpropagation derivations, then into CNNs for vision and RNNs for sequences.

The core message to remember is that deep learning’s strength lies in learning layered, non-linear features from data and improving through gradient-based optimization. With consistent effort, smart use of course resources, and a clear understanding of the fundamentals you learned today, you’ll be equipped to build models that solve meaningful, real-world problems in vision, language, and speech.