Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Neural Networks: Zero to Hero · Lecture 5 of 10 · 1:55:24

Lecture 5: Becoming a Backprop Ninja

Building makemore Part 4: Becoming a Backprop Ninja on YouTube

Study guide

What this lecture covers

The lecture answers a focused question: what actually happens inside loss.backward()? Instead of relying on PyTorch autograd, you rewrite the backward pass of the two-layer MLP with BatchNorm from the previous makemore video by hand, tensor by tensor, checking each gradient against PyTorch's own values.

It sits between the manual scalar-level autograd built in micrograd and the fully automated training loops used from here on. After watching, you can trace how a gradient flows from a loss back through cross-entropy, a linear layer, tanh, batch normalization, another linear layer, and an embedding table, and you understand why getting this right matters for debugging real networks.

Key ideas

  • Backprop as a leaky abstraction: stacking differentiable layers and calling .backward() doesn't guarantee correct or efficient training; not understanding the internals leads to subtle bugs.
  • Broadcasting implies summation backward: whenever a forward operation replicates a tensor to match shapes, the backward pass must sum the incoming gradients back down to the original shape.
  • Reuse implies addition backward: if a variable is used in multiple places in the forward pass, its gradients from each use must be added together (+=), not overwritten.
  • Shape-driven derivation: for matrix multiplies, you can recover the correct backward formula just by matching input/output shapes with transposes, without memorizing calculus rules.
  • Analytic shortcuts: deriving the gradient of cross_entropy and of BatchNorm directly on paper, rather than backpropagating through every intermediate op, gives a much shorter and faster backward pass.
  • Bessel's correction: dividing the variance sum by n - 1 instead of n gives an unbiased estimator, which the lecture prefers over PyTorch's inconsistent training/inference behavior.
  • Cross-entropy gradient intuition: the gradient on the logits is just the softmax probabilities with 1 subtracted at the correct class, acting like a push-pull force whose total is zero per row.

Walkthrough

Why hand-write backprop, and a brief history (0:00)

The lecture opens by arguing that writing the backward pass manually, even though nobody does it in practice anymore, builds the intuition needed to debug real networks. It walks through historical examples, from Hinton's 2006 restricted Boltzmann machine code in Matlab to Karpathy's own 2014 image-sentence alignment work, where gradients were computed and checked by hand before autograd engines became standard.

Starter code and the structure of the exercise (7:26)

The forward pass from the prior lecture is expanded into many small intermediate tensors (logprobs, probs, counts, logits, hpreact, etc.) so each step can be backpropagated individually. A cmp utility compares each manually computed gradient against PyTorch's .grad for exactness, closeness, and maximum difference.

Exercise 1: backpropagating the full compute graph (13:01)

This is the bulk of the lecture. Starting from dlogprobs, the derivative is worked out step by step back through probs, the softmax normalization (counts, counts_sum, counts_sum_inv), norm_logits, logit_maxes, the linear layer producing logits (deriving dh, dW2, db2 by matching shapes), tanh into hpreact, the BatchNorm gain/bias/normalized output, the first linear layer, and finally the embedding lookup table, using a loop with += to accumulate gradients for reused rows. Each derivative is checked against PyTorch before moving to the next.

Digression: Bessel's correction in BatchNorm (1:05:17)

A detour explains why the lecture divides by n - 1 rather than n when computing the batch variance, contrasting it with the original BatchNorm paper's inconsistent use of biased variance during training and unbiased variance at inference.

Exercise 2: cross-entropy loss backward pass in one step (1:26:31)

Rather than backpropagating through every atomic piece of the loss, the gradient of cross-entropy with respect to the logits is derived analytically on paper. The result: softmax(logits) with 1 subtracted at the correct-label position, scaled by 1/n. This is implemented in a few lines and shown to match PyTorch closely, and its push-pull interpretation is explained visually.

Exercise 3: batch normalization backward pass in one step (1:36:37)

The same analytical approach is applied to BatchNorm: working through the compute graph for mu, sigma^2, xhat, and y on paper, then simplifying (noting that a term vanishes because mu is the mean by construction) to arrive at a single compact formula for dhprev given dhpreact.

Exercise 4: putting it all together (1:50:02)

All derived gradients are assembled into roughly 20 lines that replace loss.backward() entirely. The network is retrained using only these manually computed gradients under torch.no_grad(), reaching the same loss and producing similar name samples as before, confirming the manual implementation is correct.

Before you watch

  • Complete the previous makemore lecture on building and training the 2-layer MLP with BatchNorm, since this lecture reuses that architecture and code unchanged.
  • Be comfortable with the micrograd exercises on scalar-level backpropagation and the chain rule.
  • Review basic matrix calculus intuition (or be ready to derive it from small 2x2 examples, as the lecture does).
  • Have the linked Jupyter/Colab notebook open to attempt each exercise before watching the solution.

Check your understanding

  1. Why does a broadcasted tensor in the forward pass require a sum in the backward pass, and why does a reused variable require an addition instead?
  2. What is the gradient of the cross-entropy loss with respect to the logits, expressed in terms of the softmax probabilities?
  3. Why does the term involving d(sigma^2)/d(mu) vanish when deriving the BatchNorm backward pass?
  4. How can you determine the correct transpose pattern for dW and dX in a linear layer's backward pass just from tensor shapes?
  5. Why does Bessel's correction (n - 1) give a better variance estimate than dividing by n when batches are small samples of a larger population?

Vocabulary

backward pass (noun)
The step where gradients are computed by moving through the network from output to input.
This lecture writes the backward pass by hand.
autograd (noun)
A system that automatically computes gradients for you, without manual derivation.
PyTorch's autograd normally computes gradients through loss.backward().
leaky abstraction (noun)
A simplified tool that hides details which sometimes still cause problems you must understand.
Backprop is called a leaky abstraction because bugs can hide underneath it.
intermediate (adjective)
Existing between the start and the end of a process, not the final result.
Many intermediate tensors are named so each step can be checked.
matching shapes (phrase)
Making sure the sizes of a tensor's dimensions line up correctly with another tensor.
You can derive a matrix multiply's gradient just by matching shapes.
transpose (verb)
To flip a matrix so its rows become columns and columns become rows.
The weight gradient uses the transpose of the input matrix.
sum backward (phrase)
Adding gradients together when the forward pass had spread one value across many copies.
Broadcasting in the forward pass requires a sum backward in the gradient.
accumulate (verb)
To add values together over multiple uses instead of replacing them.
Gradients accumulate when a variable is reused in several places.
analytic (adjective)
Solved directly with a formula, rather than step by step through a computation graph.
The analytic gradient of cross-entropy is much shorter than the full derivation.
Bessel's correction (noun)
Dividing by n minus 1 instead of n when estimating variance from a sample, to remove bias.
Bessel's correction gives an unbiased estimate of the batch variance.
unbiased estimator (noun)
A way of estimating a value that, on average, gives the correct true value.
Dividing by n - 1 makes the variance an unbiased estimator.
push-pull (adjective)
Describing forces that move in opposite directions and balance each other out.
The cross-entropy gradient has a push-pull effect on the correct and incorrect classes.
compute graph (noun)
The network of operations showing how each value was calculated from earlier ones.
The backward pass walks the compute graph from the loss to the inputs.
vanish (verb)
To become zero or disappear from a calculation.
One term in the batchnorm derivation vanishes because it sums to zero.
restricted Boltzmann machine (noun)
An older type of neural network used for learning patterns in data, popular before modern deep learning.
Hinton's restricted Boltzmann machine code used hand-derived gradients.
derive (verb)
To work out a formula step by step using logical or mathematical reasoning.
You derive the gradient of cross-entropy directly on paper.
compact (adjective)
Taking up little space; short and efficient.
The final formula for the batchnorm gradient is very compact.
detour (noun)
A brief side discussion away from the main topic.
The lecture takes a detour to explain Bessel's correction.
exactness (noun)
The quality of matching a reference value with no error at all.
The cmp utility checks each gradient for exactness against PyTorch.
atomic tensor (noun)
A single small piece of a larger computation, broken down for step-by-step checking.
The forward pass is expanded into many small atomic tensors.
embedding lookup table (noun)
A table storing one row of numbers for each possible input category.
Gradients accumulate for reused rows in the embedding lookup table.
push-pull interpretation (phrase)
A way of viewing a gradient as forces pulling toward and pushing away from an outcome.
The push-pull interpretation shows how cross-entropy nudges each class score.

Chapters

From the YouTube description

We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks.

!!!!!!!!!!!!
I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here:
https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing
!!!!!!!!!!!!

Links:
- makemore on github: https://github.com/karpathy/makemore
- jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part4_backprop.ipynb
- collab notebook: https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing
- my website: https://karpathy.ai
- my twitter: https://twitter.com/karpathy
- our Discord channel: https://discord.gg/3zy8kqD9Cp

Supplementary links:
- Yes you should understand backprop: https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b
- BatchNorm paper: https://arxiv.org/abs/1502.03167
- Bessel’s Correction: http://math.oxford.emory.edu/site/math117/besselCorrection/
- Bengio et al. 2003 MLP LM https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

Chapters:
00:00:00 intro: why you should care & fun history
00:07:26 starter code
00:13:01 exercise 1: backproping the atomic compute graph
01:05:17 brief digression: bessel’s correction in batchnorm
01:26:31 exercise 2: cross entropy loss backward pass
01:36:37 exercise 3: batch norm layer backward pass
01:50:02 exercise 4: putting it all together
01:54:24 outro

← Lecture 4: Activations, Gradients, and Batch Normalization · Lecture 6: Building a WaveNet →