Lecture 5: Becoming a Backprop Ninja

What this lecture covers

The lecture answers a focused question: what actually happens inside loss.backward()? Instead of relying on PyTorch autograd, you rewrite the backward pass of the two-layer MLP with BatchNorm from the previous makemore video by hand, tensor by tensor, checking each gradient against PyTorch's own values.

It sits between the manual scalar-level autograd built in micrograd and the fully automated training loops used from here on. After watching, you can trace how a gradient flows from a loss back through cross-entropy, a linear layer, tanh, batch normalization, another linear layer, and an embedding table, and you understand why getting this right matters for debugging real networks.

Key ideas

• Backprop as a leaky abstraction: stacking differentiable layers and calling .backward() doesn't guarantee correct or efficient training; not understanding the internals leads to subtle bugs.
• Broadcasting implies summation backward: whenever a forward operation replicates a tensor to match shapes, the backward pass must sum the incoming gradients back down to the original shape.
• Reuse implies addition backward: if a variable is used in multiple places in the forward pass, its gradients from each use must be added together (+=), not overwritten.
• Shape-driven derivation: for matrix multiplies, you can recover the correct backward formula just by matching input/output shapes with transposes, without memorizing calculus rules.
• Analytic shortcuts: deriving the gradient of cross_entropy and of BatchNorm directly on paper, rather than backpropagating through every intermediate op, gives a much shorter and faster backward pass.
• Bessel's correction: dividing the variance sum by n - 1 instead of n gives an unbiased estimator, which the lecture prefers over PyTorch's inconsistent training/inference behavior.
• Cross-entropy gradient intuition: the gradient on the logits is just the softmax probabilities with 1 subtracted at the correct class, acting like a push-pull force whose total is zero per row.

Walkthrough

Why hand-write backprop, and a brief history (0:00)

The lecture opens by arguing that writing the backward pass manually, even though nobody does it in practice anymore, builds the intuition needed to debug real networks. It walks through historical examples, from Hinton's 2006 restricted Boltzmann machine code in Matlab to Karpathy's own 2014 image-sentence alignment work, where gradients were computed and checked by hand before autograd engines became standard.

Starter code and the structure of the exercise (7:26)

The forward pass from the prior lecture is expanded into many small intermediate tensors (logprobs, probs, counts, logits, hpreact, etc.) so each step can be backpropagated individually. A cmp utility compares each manually computed gradient against PyTorch's .grad for exactness, closeness, and maximum difference.

Exercise 1: backpropagating the full compute graph (13:01)

This is the bulk of the lecture. Starting from dlogprobs, the derivative is worked out step by step back through probs, the softmax normalization (counts, counts_sum, counts_sum_inv), norm_logits, logit_maxes, the linear layer producing logits (deriving dh, dW2, db2 by matching shapes), tanh into hpreact, the BatchNorm gain/bias/normalized output, the first linear layer, and finally the embedding lookup table, using a loop with += to accumulate gradients for reused rows. Each derivative is checked against PyTorch before moving to the next.

Digression: Bessel's correction in BatchNorm (1:05:17)

A detour explains why the lecture divides by n - 1 rather than n when computing the batch variance, contrasting it with the original BatchNorm paper's inconsistent use of biased variance during training and unbiased variance at inference.

Exercise 2: cross-entropy loss backward pass in one step (1:26:31)

Rather than backpropagating through every atomic piece of the loss, the gradient of cross-entropy with respect to the logits is derived analytically on paper. The result: softmax(logits) with 1 subtracted at the correct-label position, scaled by 1/n. This is implemented in a few lines and shown to match PyTorch closely, and its push-pull interpretation is explained visually.

Exercise 3: batch normalization backward pass in one step (1:36:37)

The same analytical approach is applied to BatchNorm: working through the compute graph for mu, sigma^2, xhat, and y on paper, then simplifying (noting that a term vanishes because mu is the mean by construction) to arrive at a single compact formula for dhprev given dhpreact.

Exercise 4: putting it all together (1:50:02)

All derived gradients are assembled into roughly 20 lines that replace loss.backward() entirely. The network is retrained using only these manually computed gradients under torch.no_grad(), reaching the same loss and producing similar name samples as before, confirming the manual implementation is correct.

Before you watch

• Complete the previous makemore lecture on building and training the 2-layer MLP with BatchNorm, since this lecture reuses that architecture and code unchanged.
• Be comfortable with the micrograd exercises on scalar-level backpropagation and the chain rule.
• Review basic matrix calculus intuition (or be ready to derive it from small 2x2 examples, as the lecture does).
• Have the linked Jupyter/Colab notebook open to attempt each exercise before watching the solution.

Check your understanding

1. Why does a broadcasted tensor in the forward pass require a sum in the backward pass, and why does a reused variable require an addition instead?
2. What is the gradient of the cross-entropy loss with respect to the logits, expressed in terms of the softmax probabilities?
3. Why does the term involving d(sigma^2)/d(mu) vanish when deriving the BatchNorm backward pass?
4. How can you determine the correct transpose pattern for dW and dX in a linear layer's backward pass just from tensor shapes?
5. Why does Bessel's correction (n - 1) give a better variance estimate than dividing by n when batches are small samples of a larger population?