Lecture 3: Backpropagation and Neural Networks

What this lecture covers

This lecture explains how neural networks are actually trained: by computing gradients of a loss function and adjusting parameters in the downhill direction. It starts by formalizing the layered, matrix-multiplication view of a neural network introduced with the location classifier example in Lecture 2, and surveys the activation functions (sigmoid, tanh, ReLU, and newer variants) used to make those layers nonlinear.

The bulk of the lecture is a hands-on introduction to matrix calculus — gradients, Jacobians, and the chain rule applied to vectors and matrices — followed by an explanation of backpropagation as an efficient, general algorithm for computing those gradients on a computation graph. By the end, you can explain why nonlinear activation functions are required, derive the gradient of a simple neural network layer by hand, and describe what a deep learning framework is doing when it runs a forward and backward pass.

Key ideas

• Layered neural network: computed as H = f(WX + B), a matrix multiplication and bias addition followed by an elementwise nonlinearity.
• Activation functions: nonlinear functions like the logistic sigmoid, tanh, and ReLU that give a neuron a gradient to learn from and let the network approximate complex, non-linear functions.
• Why nonlinearity is essential: stacking multiple linear (matrix-multiply) layers with no nonlinearity in between collapses mathematically into a single linear transform, giving no extra representational power.
• Jacobian: the matrix of partial derivatives of every output of a function with respect to every input, used when composing multi-input, multi-output functions with the chain rule.
• Shape convention: the practical rule that a computed gradient should be reshaped to match the shape of the parameter it corresponds to, so it can be directly subtracted during gradient descent.
• Computation graph: a representation of a function as source nodes (inputs) and operation nodes, used to run a forward pass (compute values) and a backward pass (compute gradients).
• Backpropagation: repeatedly applying the chain rule from the output back toward the inputs, reusing already-computed "upstream" gradients so that no part of the calculation is repeated.
• Gradient checking: a numerical technique for verifying a backward-pass implementation by comparing it to a finite-difference estimate of the slope.

Walkthrough

From logistic units to matrix-layer neural networks (5:11)

Manning reframes the individually wired neurons from the previous lecture as a regular, layered structure: the output of one layer is combined with weights and a bias into a vector Z = WX + B, then passed through a nonlinearity to produce the next layer's values. This matrix-multiplication view is what the location-detection example from Lecture 2 actually computes, and it is the standard building block used throughout the course.

Where activation functions come from (8:13)

The lecture traces the history of activation functions from 1940s threshold units (which have no gradient and therefore cannot be learned by gradient descent) through the sigmoid and tanh functions, to the rectified linear unit (ReLU), which is zero for negative inputs and has a constant gradient of one for positive inputs. Despite being "dead" for half its range, ReLU became the default choice for years because it gave efficient gradient flow, and more recent Transformer models often use smoother ReLU-like variants such as Swish and GELU.

Why nonlinearities are necessary (14:20)

Neural networks are framed as function approximators. Composing multiple pure matrix multiplications is mathematically equivalent to a single matrix multiplication, so stacking linear layers alone adds no representational power, even though it can still affect what is learned. Nonlinear activation functions are what allow a multi-layer network to represent genuinely complex, curved functions rather than only straight decision boundaries.

Matrix calculus: gradients, Jacobians, and the chain rule (20:23)

Manning's core message is that multivariable calculus works "just like single-variable calculus, but with matrices." He builds up from a single-variable derivative, to a gradient (a vector of partial derivatives for a function with several inputs), to a Jacobian (a matrix of partial derivatives for a function with several inputs and several outputs). Composing functions, as neural network layers do, means multiplying their Jacobians together in the same way that single-variable derivatives multiply under the chain rule.

Computing gradients for a neural network layer (26:32)

Working through the small location-classifier network, the lecture derives the gradient of the score with respect to the bias, weight matrix, and inputs. A key efficiency idea appears here: the "upstream" part of the gradient calculation (referred to as delta, or the error signal) is shared across the derivatives with respect to different parameters, so it should be computed once and reused rather than recalculated for each parameter. The lecture also introduces the shape convention, reshaping derivatives so they match the shape of the parameter tensors they update.

Backpropagation as a computation graph (44:55)

Manning represents the network as a computation graph with input nodes and operation nodes, and shows that a forward pass simply evaluates the function while a backward pass, starting from a gradient of one at the output, multiplies each node's local gradient by its upstream gradient to get a downstream gradient. Working through a small worked example (f(x,y,z) = (x+y) * max(y,z)), he shows how addition distributes gradient equally, max routes gradient only to the larger input, and multiplication swaps gradients between the two multiplied values, and that gradients from multiple paths into the same variable are summed.

Gradient checking (1:08:34)

To verify a backward-pass implementation is correct, Manning describes numerical gradient checking: estimating the slope at a point using a small step H (typically around 1e-4) and comparing (f(x+H) - f(x-H)) / (2H) to the analytically computed gradient. This two-sided estimate is more accurate than a one-sided one, but it is far too slow to use for actual training, so it is reserved for checking that a hand-written gradient computation is correct.

Before you watch

• Watch Lecture 1 and Lecture 2 first: this lecture builds directly on the word vector and simple classifier examples introduced there.
• Comfort with basic linear algebra (matrix multiplication, dot products) and single-variable derivatives makes the matrix calculus section much easier to follow.

Check your understanding

1. Why does stacking multiple layers of pure matrix multiplication fail to add representational power, and what fixes that?
2. What is a Jacobian, and why is it needed once a function has multiple inputs and multiple outputs?
3. In the derivation of the neural network layer's gradient, what is the "upstream gradient" (delta), and why is it computed only once?
4. How do addition, max, and multiplication operations each route gradients differently in a computation graph during backpropagation?
5. Why is numerical gradient checking useful for debugging, even though it is too slow to use during actual training?