Lecture 4: Introduction to Neural Networks

What this lecture covers

The previous lecture showed how to compute gradients analytically for a simple linear classifier, but that approach doesn't scale to the far more complex functions used in deep learning. This lecture introduces backpropagation: representing any function as a computational graph and computing gradients by recursively applying the chain rule, one simple local step at a time. It then uses this machinery to introduce neural networks as multiple linear layers stacked with non-linearities between them.

This is the fourth lecture and builds on the loss functions and gradient descent from Lecture 3. After watching, you'll be able to build a computational graph for a function, run backpropagation through it by hand, and explain how a two-layer neural network differs from the single linear classifier covered earlier in the course.

Key ideas

• Computational graph: any function can be represented as a graph of nodes, where each node is a simple operation (addition, multiplication, exponential, and so on) with known inputs, outputs, and local gradients.
• Backpropagation: starting from the end of the graph and working backward, each node takes the upstream gradient flowing in and multiplies it by its own local gradient (the chain rule) to produce the gradient to pass further back.
• Local computation: each node only needs to know its immediate inputs, output, and local gradient; it doesn't need to understand the rest of the graph, which is what makes backprop tractable for arbitrarily complex functions.
• Gate intuitions: an add gate distributes the upstream gradient equally to both inputs; a max gate routes the full gradient to whichever input was the maximum and zero to the other; a multiply gate scales the upstream gradient by the value of the other input.
• Gradients add at branches: when one node's output feeds into multiple downstream nodes, the gradients flowing back from each are summed.
• Jacobians for vectors: when inputs and outputs are vectors, the local "gradient" becomes a Jacobian matrix; in practice these are often sparse (for example, diagonal for element-wise operations) so they don't need to be formed explicitly.
• Forward/backward API: deep learning frameworks implement each computational node as a class with a forward method (compute output, cache needed values) and a backward method (apply the chain rule using cached values).
• Neural networks as stacked layers: a two-layer neural network is f = W2 * max(0, W1*x), replacing the single template per class in a linear classifier with multiple templates (W1) that get combined (W2), and non-linearities between layers are essential or the layers collapse into one linear function.
• Loose biological analogy: neurons combining input signals and firing resemble the weighted-sum-plus-activation-function structure of a network node, but the lecture stresses this analogy is approximate and real neurons are far more complex.

Walkthrough

Recap and computational graphs (4:11)

After brief administrative notes, the lecture recaps the previous lecture's pipeline: a score function, a loss function combining data loss and regularization, and gradient descent using either numerical (finite-difference) or analytic gradients. It then introduces computational graphs as a way to represent any function, however complex, as a network of simple computation nodes, and states that this representation enables backpropagation, a recursive application of the chain rule that computes the gradient with respect to every variable in the graph.

A simple backpropagation example (5:11)

Using f(x, y, z) = (x + y) * z, the lecture builds the computational graph, performs a forward pass with concrete numbers, then works backward node by node. At each node, it applies the chain rule: the gradient with respect to an input equals the upstream gradient (the gradient of the final output with respect to that node's output) multiplied by the node's local gradient. The example shows how df/dy and df/dx are found without ever writing a single global expression, since each node only reasons about its own local inputs and outputs.

A more complex example: the sigmoid gate (18:31)

The lecture works through a larger function, f(w, x) = 1/(1 + e^(-(w0*x0 + w1*x1 + w2))), breaking it into a chain of multiply, add, negate, exponential, add-one, and divide nodes. Backpropagating step by step through each gate produces the same result as deriving the whole expression analytically by hand, but using only simple local derivatives at each step. The lecture also shows that nodes can be grouped at any granularity, demonstrating that treating the entire sigmoid expression as one node (using its known derivative, sigmoid(x) * (1 - sigmoid(x))) gives an identical gradient, illustrating a tradeoff between how much calculus to do upfront versus how many graph nodes to track.

Patterns in backpropagation and modular implementation (31:17)

The lecture highlights recognizable patterns at common gate types: an add gate acts as a "gradient distributor" (passing the same upstream gradient to both branches), a max gate acts as a "gradient router" (routing the full gradient only to the branch that was the maximum), and a multiply gate acts as a "gradient switcher/scaler" (scaling the upstream gradient by the other input's value). It also covers the rule that gradients sum when a node's output branches into multiple downstream paths. The lecture then shows how this maps directly onto code: each gate is implemented with a forward function (computing the output and caching values needed later) and a backward function (applying the chain rule using those cached values), which is how real frameworks like Caffe structure their layers.

Vectorized gradients and Jacobians (37:44)

When inputs and outputs become vectors instead of scalars, local gradients become Jacobian matrices, one partial derivative per pair of input and output dimensions. The lecture illustrates with an element-wise max operation on a 4096-dimensional vector: the full Jacobian would be 4096-by-4096 (and much larger once batched), but because the operation is element-wise, the Jacobian is diagonal and never needs to be built explicitly. A worked example with f(x, W) = ||Wx||^2 shows deriving gradients with respect to both x and W using this same chain-rule process, with the practical checkpoint that a gradient should always have the same shape as the variable it corresponds to.

From linear classifiers to neural networks (55:24)

Having established how to compute gradients for arbitrary functions, the lecture introduces neural networks as stacked linear layers separated by non-linearities, for example a two-layer network f = W2 * max(0, W1*x). It explains why non-linearities are required: stacking linear layers without them collapses into a single linear function. Revisiting the earlier idea that each row of a linear classifier's weight matrix is a single template per class, the lecture shows how a two-layer network lets an intermediate layer learn multiple templates (for example, both a left-facing and right-facing horse), with a second weight matrix combining them into the final class score, addressing the single-template limitation from Lecture 2.

Biological inspiration and network terminology (1:04:38)

The lecture draws a loose analogy between artificial network nodes and biological neurons: dendrites receiving and integrating signals, a cell body applying something like an activation function, and axons carrying the output onward, with the ReLU activation noted as one of the closer (though still imperfect) matches to real neuron firing behavior. It stresses that real neurons are far more complex than this analogy suggests. The lecture closes by clarifying naming conventions (a "two-layer" network has two weight matrices and one hidden layer) and noting that these layers can be evaluated efficiently as vectorized matrix multiplications, setting up the next lecture's move to convolutional neural networks.

Before you watch

• Watch Lectures 2 and 3 first; this lecture assumes familiarity with the linear classifier, loss functions, and gradient descent.
• Comfort with the multivariable chain rule and basic matrix/vector operations will make the backpropagation examples easier to follow.
• Recall the "one template per class" limitation of linear classifiers from Lecture 2, since this lecture directly addresses it.

Check your understanding

1. In backpropagation, what two things does a node need to compute the gradient it passes backward, and why does it not need information from the rest of the graph?
2. What is the intuitive behavior of an add gate, a max gate, and a multiply gate during the backward pass?
3. Why is it impractical to explicitly form the Jacobian matrix for an element-wise operation on a large vector, and how is this avoided in practice?
4. Why does stacking linear layers without a non-linearity between them fail to produce a more expressive function?
5. How does a two-layer neural network address the "single template per class" limitation of a linear classifier?