Lecture 4: Activations, Gradients, and Batch Normalization

What this lecture covers

This lecture continues the makemore series, staying with the multilayer perceptron (MLP) built in the previous video instead of moving straight to recurrent networks. It answers a more fundamental question first: why are deep neural networks hard to optimize, and how do the activations and gradients flowing through a network reveal what's going wrong? Understanding this is presented as a prerequisite for later architectures like RNNs, which the lecture argues are hard to train precisely because of these activation and gradient dynamics.

By the end, you can recognize a poorly initialized network from its loss curve and activation histograms, fix initialization using principled scaling (Kaiming-style initialization), and understand what batch normalization does, why it works, and its costs. The lecture also refactors the code into PyTorch-style modules (Linear, BatchNorm1d, Tanh) and introduces diagnostic plots for activations, gradients, and parameter update ratios that are used to judge whether a network is training well.

Key ideas

• Expected initial loss: at initialization, with no reason to prefer any output, the loss should be close to -log(1/27) (uniform distribution over 27 characters); a much higher value signals the network is confidently wrong.
• The hockey-stick loss curve: an initial loss far above the expected value produces a loss curve that drops sharply in the first iterations, because early training is wasted just shrinking down oversized logits and weights instead of learning.
• Saturated tanh: when pre-activation values are too large in magnitude, most tanh outputs sit near -1 or 1, where the local gradient (1 - t^2) is near zero, so gradients are killed during backpropagation.
• Dead neurons: a neuron whose output never leaves the flat, saturated region for any training example never receives a gradient and stops learning permanently, a risk shared by tanh, sigmoid, and ReLU.
• Kaiming-style initialization: scaling a layer's weights by a gain divided by the square root of the fan-in keeps activations from shrinking to zero or exploding as they pass through many layers; the appropriate gain depends on the nonlinearity used.
• Batch normalization: standardizing each hidden layer's pre-activations to zero mean and unit standard deviation (per batch), then rescaling with a learned gain and bias, keeps activations well-behaved without hand-tuning every layer's initialization.
• Batchnorm's side effects: normalizing over a batch couples examples together mathematically, which acts as an accidental regularizer but also introduces bugs and complications, including the need for running mean/standard deviation estimates for inference on single examples.
• Update-to-data ratio: tracking how large each parameter's update is relative to its own scale (roughly 10^-3 is a good target) is a more informative diagnostic than the loss curve alone for spotting a learning rate that's too high or too low.

Walkthrough

Diagnosing the too-high initial loss (4:19)

Starting from a cleaned-up version of the previous lecture's MLP, the lecture points out that the very first training loss is about 27, far above the roughly 3.29 that a uniform distribution over 27 characters would produce. A small four-character toy example demonstrates that when logits are all near zero, the softmax output is close to uniform and the loss matches the expected value, but as logits take on extreme values the model becomes "confidently wrong" and the loss spikes, sometimes even to infinity. The fix is to shrink the final layer's weights and zero out its biases at initialization, so the initial logits start near zero.

Fixing the saturated tanh (12:59)

Even after fixing the output layer, a histogram of the hidden layer's tanh activations shows most values pinned at -1 or 1. Because the tanh backward pass multiplies the incoming gradient by 1 - t^2, saturated neurons block gradient flow almost entirely, and a neuron saturated for every training example becomes a permanently dead neuron. Scaling down the hidden layer's weights so the pre-activations land in a smaller range fixes the saturation, and retraining with both fixes applied improves the validation loss compared to the original run.

Kaiming initialization (27:53)

Rather than tuning scaling factors by trial and error, the lecture derives a principled approach: multiplying a unit-Gaussian input by weights increases the output's standard deviation, and dividing by the square root of the fan-in restores it to one. This matches the "Kaiming init" paper (Delving Deep into Rectifiers), which adds a nonlinearity-specific gain to compensate for how much a given activation function squashes its input; the gain for tanh is 5/3, and PyTorch's torch.nn.init.kaiming_normal_ implements the same idea. The lecture notes that modern techniques like residual connections, normalization layers, and better optimizers have made precise initialization less critical than it was when this paper was published.

Batch normalization (40:40)

Batch normalization, introduced by a Google team in 2015, standardizes each hidden layer's pre-activations across the batch (subtracting the batch mean, dividing by the batch standard deviation), then applies a learned per-neuron gain and bias so the network can still shift the distribution as needed during training. Because normalizing over a batch means one example's activations depend on every other example in that batch, the lecture describes this coupling as both a useful side-effect regularizer and a common source of bugs. To support inference on single examples, the layer maintains running estimates of the mean and standard deviation, updated during training via exponential moving average, so no separate calibration pass is needed. The lecture also shows that biases become redundant in a layer immediately followed by batchnorm, since they get subtracted out by the normalization, and points out batchnorm's real-world use inside a ResNet's convolution-batchnorm-ReLU blocks.

Rebuilding the network as reusable PyTorch-style modules (1:18:35)

The code is refactored into Linear, BatchNorm1d, and Tanh classes modeled closely on PyTorch's own torch.nn API, including a .training flag that changes batchnorm's behavior between training (using batch statistics) and evaluation (using running statistics). Stacking these into a deeper six-layer network with about 46,000 parameters sets up the diagnostic plots covered next.

Diagnostic plots for activations, gradients, and updates (1:26:51)

With the deeper network, the lecture plots histograms of each tanh layer's forward activations, backward gradients, and parameter statistics, along with the percentage of saturated neurons per layer. Removing the tanh gain shows activations collapsing toward zero across layers; too large a gain saturates them instead. Removing all nonlinearities (a pure linear sandwich) shows that the correct gain is exactly 1 for a stable stack, illustrating how fragile manual calibration is without normalization. The update-to-data ratio plot is introduced as a way to see, layer by layer, whether parameters are being updated too aggressively or too slowly relative to their own scale, with roughly 10^-3 on a log10 scale used as a rough target.

Making batchnorm robust to gain choices (1:46:04)

Adding batchnorm layers back into the six-layer network shows that activations and gradients stay well-behaved even when the linear layers' gain is set far from its "correct" value, or when fan-in scaling is removed entirely, since the explicit normalization compensates. The main remaining sensitivity is that the update-to-data ratios can shift when activation scales change substantially, meaning the learning rate may still need retuning. The lecture closes by summarizing batch normalization's role in enabling deep network training, previewing that recurrent networks (effectively very deep networks once unrolled) will make these lessons about activations and gradients especially important.

Before you watch

• Watch the previous makemore lecture on the MLP model first, since this one reuses its dataset, embedding table, and training loop without re-explaining them.
• Review how tanh and its derivative behave, ideally from the micrograd lecture, since the gradient-killing argument depends on it.
• Familiarity with mean, variance, and standard deviation will help with the batch normalization derivation.

Check your understanding

1. Why should the loss at initialization be close to -log(1/27) for this character-level model, and what does a much higher initial loss indicate?
2. How does a saturated tanh neuron affect gradient flow during backpropagation, and what makes a neuron "dead"?
3. What does dividing weights by the square root of the fan-in achieve during initialization, and why does the nonlinearity's gain matter?
4. What problem does batch normalization solve, and what complication does it introduce for running inference on a single example?
5. Why is the update-to-data ratio considered more informative than just watching the loss curve when tuning a learning rate?