Lecture 3 (Part II): Manual Neural Networks

What this lecture covers

This is the second half of Lecture 3, continuing directly from Part I's introduction of neural networks as learnable feature extractors. Here the lecture derives backpropagation by hand: first the gradients for a simple two-layer network, then the general recurrence for an arbitrary L-layer feed-forward network, all worked out in real matrix arithmetic rather than toy scalar notation.

By the end, you'll understand backpropagation as the forward pass (computing each layer's activations Z_i) followed by a backward pass (computing gradient terms G_i that flow from the output back to each layer), with intermediate results cached along the way. The lecture closes by reframing this as a "vector-Jacobian product" that each layer must implement, setting up the shift to automatic differentiation in the next lecture, after which manual derivations like this one won't be needed again.

Key ideas

• Backpropagation is just the gradient step: the hypothesis class changes from linear to neural network, but the loss (cross-entropy) and optimizer (SGD) stay the same as in softmax regression; only computing the gradient with respect to the network's weights differs.
• Two-layer gradients: for h_theta(x) = sigma(X W_1) W_2, the gradient with respect to W_2 is sigma(X W_1)^T (S - I_y), where S is the softmax of the network output; the gradient with respect to W_1 additionally involves sigma'(X W_1) and an elementwise product.
• The backward gradient term G_i: defined as the gradient of the loss with respect to layer Z_i, computed via a backward recurrence G_i = (G_(i+1) elementwise* sigma'(Z_i W_i)) W_i^T, reusing terms computed for later layers.
• Forward and backward passes: the forward pass computes each Z_i in sequence from the input; the backward pass computes each G_i in reverse order, and the per-layer parameter gradient is Z_i^T (G_(i+1) elementwise* sigma'(Z_i W_i)).
• Caching cost: because backward-pass gradients need the forward-pass activations Z_i, none of them can be discarded early, so backpropagation trades extra memory for computational efficiency (roughly double the cost of a forward pass alone, plus stored activations).
• Vector-Jacobian product: each layer's job in backpropagation reduces to multiplying an incoming backward gradient by its own local derivative; this modular operation is what automatic differentiation systems implement generically per layer type.

Walkthrough

Setting up the two-layer gradient problem (0:09)

The lecture recaps that only the hypothesis class changed from softmax regression to neural networks, so the task is computing gradients with respect to W_1 and W_2 for h_theta(x) = sigma(X W_1) W_2, using the same "treat everything as scalar, then fix sizes" trick from the previous lecture.

Deriving the two-layer gradients (2:11)

The gradient with respect to W_2 turns out structurally identical to the softmax regression gradient, just with sigma(X W_1) standing in for X. The gradient with respect to W_1 requires an extra chain-rule step through the nonlinearity, producing the elementwise product term sigma'(X W_1); the lecture works through matching matrix dimensions to arrive at X^T ((S - I_y) W_2^T elementwise* sigma'(X W_1)).

Generalizing to an L-layer network (19:39)

The lecture extends the derivation to arbitrary depth, noting that the chain rule produces a long product of partial derivatives across layers, many of which repeat across different weight gradients. This motivates defining G_i, the gradient of the loss with respect to layer Z_i, and showing the simple recurrence relating G_i to G_(i+1).

Computing the real matrix-form gradients (28:07)

Moving from the "pretend everything is scalar" derivation to actual matrix shapes, the lecture derives the real backward update G_i = (G_(i+1) elementwise* sigma'(Z_i W_i)) W_i^T and the parameter gradient Z_i^T (G_(i+1) elementwise* sigma'(Z_i W_i)), matching dimensions carefully at each step.

The backpropagation algorithm: forward and backward passes (38:05)

The lecture assembles the full algorithm: a forward pass computing each Z_i from the input, and a backward pass initializing the last gradient term from the softmax output minus the one-hot label, then iterating the G_i recurrence backward through the layers while computing each parameter's gradient. It notes the memory cost of caching forward activations for use in the backward pass.

The vector-Jacobian product and what's next (43:19)

The lecture reframes each layer's role as multiplying an incoming backward gradient by its own local derivative, a general operation called the vector-Jacobian product. This modular view is what makes automatic differentiation possible, and the lecture closes by saying this is the last time manual backpropagation derivations like this will be needed in the course.

Before you watch

• Watch Lecture 3 Part I first; this lecture continues directly from its two-layer network definition.
• Review Lecture 2's gradient derivation for softmax regression, since this lecture reuses its "scalar trick, then match matrix sizes" method repeatedly.
• Comfort with the chain rule and matrix dimension matching is essential to following the derivations.

Check your understanding

1. Why is the gradient with respect to W_2 in the two-layer network structurally similar to the softmax regression gradient from the previous lecture?
2. What is the recurrence relating G_i to G_(i+1), and why is it useful to compute gradients this way rather than from scratch for each layer?
3. Why must backpropagation cache the forward-pass activations Z_i rather than discarding them once computed?
4. What is a vector-Jacobian product, and why does the lecture say this concept makes automatic differentiation possible?