Lecture 11: Backprop and Improving Neural Networks

What this lecture covers

This is the second of two CS229 lectures on neural networks. It picks up where the previous lecture stopped, deriving the backpropagation equations for the three-layer cat-classification network by hand using the chain rule, and then turns to practical techniques for making networks actually train well: choosing activation functions, normalizing inputs, initializing weights, and using mini-batch gradient descent with momentum.

The lecture assumes you've already seen the network's forward propagation and its notation for layers, weights, biases and activations. After watching, you should be able to derive a weight update using the chain rule and shape analysis, explain why sigmoid and tanh activations can cause vanishing gradients while ReLU generally avoids the problem, describe why inputs are normalized and weights initialized in proportion to layer size, and explain the tradeoffs among batch, stochastic and mini-batch gradient descent.

Key ideas

• Backpropagation is repeated chain rule: the derivative of the loss with respect to an early layer's weights is built by multiplying together derivatives already computed for later layers, avoiding redundant computation.
• Caching forward-pass values: intermediate activations and linear outputs (a and z at each layer) are stored during forward propagation because backpropagation needs them again.
• Shape analysis disciplines the chain rule: matching matrix and vector dimensions (transposing where needed) is how you verify a chain-rule derivative is correct, especially in higher dimensions where the "obvious" scalar-calculus answer needs adjustment.
• Activation choice affects trainability: sigmoid and tanh saturate for large-magnitude inputs, driving gradients toward zero and slowing learning in early layers; ReLU keeps a gradient of 1 for positive inputs and largely avoids this vanishing-gradient problem.
• Activations are what give depth its power: a network built entirely from linear activations collapses mathematically into a single linear function, no matter how many layers it has.
• Input normalization reshapes the loss surface: centering and scaling inputs (using statistics from the training set only) makes the loss contours more circular, which speeds up gradient descent.
• Initialization scale matters: weights are typically initialized proportional to 1/sqrt(n) (or 2/n for ReLU) of the number of inputs to a layer, to prevent the outputs of very deep networks from exploding or vanishing; initialization is also randomized to avoid a symmetry problem where neurons all learn the same thing.
• Mini-batch gradient descent and momentum: mini-batches trade off the vectorization speed of full-batch gradient descent against the fast, noisy updates of stochastic gradient descent, and momentum smooths the update direction by averaging past gradients.

Walkthrough

Recap and deriving dJ/dW3 by hand (1:05)

The lecture briefly recaps the three-layer cat-classification network and its logistic loss, then computes the derivative of the loss with respect to the last layer's weights, W3, directly. Working through the composition of the logarithm, sigmoid and linear parts term by term, and checking matrix shapes at each step (noting where a transpose is required), the derivation simplifies to (a3 - y) * a2^T. This concrete, fully worked derivative serves as the building block for the more efficient chain-rule approach used for earlier layers.

Using the chain rule to backpropagate to W2 and W1 (19:13)

Rather than repeating the full derivation for W2, the lecture shows that most of the terms needed were already computed while finding the derivative for W3. By decomposing dL/dW2 into a chain of simpler derivatives (loss with respect to a3, a3 with respect to z3, z3 with respect to a2, and so on) and reusing cached values, the earlier result can be extended with only a couple of new terms. The lecture stresses choosing the correct path through the chain rule, since taking a derivative with respect to a term that isn't connected leads to a dead end, and again emphasizes shape analysis to get element-wise versus matrix products right. Caching forward-pass values is explained as what makes this reuse possible without recomputing the forward pass.

Choosing activation functions (34:38)

The lecture compares sigmoid, ReLU and tanh, stating each function and its derivative. Sigmoid and tanh both saturate at large positive or negative inputs, meaning their gradient approaches zero there, which slows or stalls learning in earlier layers during backpropagation. ReLU avoids this for positive inputs, where its gradient is exactly 1, which is why it is used as the default hidden-layer activation in most modern networks. A separate demonstration shows that if every activation were the identity function, the entire network's forward pass reduces algebraically to a single linear transformation of the input, which is why nonlinear activations are essential to a network's expressive power.

Normalizing inputs and initializing weights (47:53)

To reduce saturation and speed up training, inputs are normalized by subtracting the training set's mean and dividing by its standard deviation, reshaping the loss surface so gradient descent moves more directly toward the minimum; the same mean and standard deviation computed on the training set must also be applied to the test set. The lecture then covers weight initialization schemes, motivated by the idea that a neuron's linear output is a sum over many weighted inputs, so more inputs call for smaller individual weights to keep that sum in a reasonable range. Common schemes initialize weights proportional to sqrt(1/n) (for sigmoid or tanh) or sqrt(2/n) (for ReLU), where n is the number of inputs to the layer, and Xavier/Glorot-style initialization additionally accounts for the number of outputs. Randomizing the initial weights (rather than using a fixed value) is also necessary to break symmetry between neurons.

Vanishing and exploding gradients (53:10)

Using a simplified deep network with identity activations and zero biases, the lecture shows that the output becomes a product of many weight matrices, and if those matrices are consistently a little larger than the identity, the product (and the output) explodes exponentially with depth; if they are consistently a little smaller, the product vanishes toward zero. The same effect happens with gradients during backpropagation, which motivates keeping weights close to a scale where this repeated multiplication stays stable, tying directly back to the initialization schemes just introduced.

Mini-batch gradient descent and momentum (65:19)

The lecture contrasts full-batch gradient descent, which vectorizes computation efficiently but is slow per update on large datasets, with stochastic gradient descent, whose updates are fast but noisy. Mini-batch gradient descent splits the training set into fixed-size batches and updates parameters after each batch, trading off vectorization speed against update frequency. Momentum is then introduced as a way to smooth the path gradient descent takes across a loss surface: instead of using only the current gradient, momentum maintains a "velocity" that averages recent updates, which dampens oscillation in steep directions while still making rapid progress in flatter ones, at the cost of one extra variable to track.

Before you watch

• Watch the previous lecture on neural network forward propagation, since this lecture builds directly on its notation for layers, weights, biases and activations.
• Be comfortable with the chain rule from multivariable calculus and with basic matrix-vector shape rules (including when a transpose is required).
• Review logistic regression's loss function and gradient descent update rule, since both reappear here in the neural network setting.

Check your understanding

1. Why does computing the derivative for W3 first, rather than W1, make the backpropagation process more efficient?
2. Why is caching forward-propagation values necessary for backpropagation to work efficiently?
3. Why do sigmoid and tanh activations tend to produce vanishing gradients, and how does ReLU avoid this for positive inputs?
4. Why does a network built entirely from linear activations behave the same as plain linear regression, regardless of depth?
5. What tradeoff does mini-batch gradient descent make between full-batch and stochastic gradient descent, and how does momentum change the direction of an update?