Lecture 9: Normalization and Regularization

What this lecture covers

The lecture asks why deep networks are hard to train reliably and what tricks make them easier. It follows on from earlier lectures on building a neural network library, and shows that initialization alone does not solve training instability: badly scaled weights either blow up into NaN values or make no progress at all, even after many gradient steps.

The lecture then introduces normalization layers (layer norm and batch norm) as an explicit fix for unstable activations, and regularization (L2 weight decay and dropout) as a way to keep overparameterized networks from overfitting. It closes with a case study on batch norm showing that even widely used techniques are not fully explained theoretically. After watching, you should be able to explain what layer norm and batch norm compute, implement L2 regularization and dropout correctly with gradient-based optimizers, and describe why these techniques interact with initialization and optimization.

Key ideas

• Initialization variance matters: for ReLU networks, weights initialized with variance 2/n keep activation and gradient norms stable across depth; slightly larger or smaller variances cause exploding or vanishing behavior in deep networks.
• Layer norm: normalizes each example's activation vector to mean 0 and variance 1 at every layer, which fixes exploding/vanishing activations but can make plain fully connected networks harder to optimize.
• Batch norm: normalizes each feature (column) across the examples in a mini-batch rather than each example (row); widely used but makes predictions for one example depend on the other examples in its batch during training.
• Running statistics at test time: batch norm tracks exponential moving averages of the mean and variance during training and uses those, not batch statistics, at test time.
• L2 regularization (weight decay): adds lambda/2 * sum of squared weight norms to the loss, which shrinks weights toward zero on every update; it is usually implemented as a property of the optimizer, not the loss.
• Dropout: randomly zeroes each activation with probability p during training and rescales the rest by 1/(1-p); best understood as a stochastic approximation of the layer's activations, similar to how SGD approximates gradient descent using mini-batches.
• Implicit vs explicit regularization: SGD itself, combined with a given weight initialization, implicitly limits which functions a network can reach, separate from explicit regularizers like weight decay or dropout.
• Theoretical uncertainty: batch norm was originally explained as reducing "internal covariate shift," but later papers dispute this and instead point to smoother optimization landscapes or robustness to distribution shift, and no explanation is fully settled.

Walkthrough

Why initialization alone is not enough (1:01)

The lecture revisits weight initialization with variance c/n and shows, using a 50-layer network on MNIST, that only c=2 keeps activation and gradient norms roughly constant across depth. Smaller variance causes activations to shrink toward zero and training to stall; larger variance causes activations and gradients to blow up, producing NaN losses. A surprising observation follows: after training, the norm of the weights at each layer is nearly indistinguishable from their value at initialization, even though the network trains successfully — initialization has a lasting effect on a deep network's behavior.

Layer normalization (12:13)

Rather than relying on careful initialization, the lecture proposes adding a layer that forces each example's activations to have mean 0 and variance 1, computed empirically and divided by the standard deviation plus a small epsilon to avoid division by zero. This stabilizes activation and gradient norms regardless of initialization and is standard in transformer architectures, but can make plain fully connected networks harder to train because it removes potentially useful information carried in the relative norms of different examples.

Batch normalization (23:45)

Viewing activations as a matrix with examples as rows, batch norm normalizes columns (features) across a mini-batch instead of rows (examples). This preserves per-example norm differences while still stabilizing the network, but introduces a dependency between examples in the same batch during training. To avoid using batch-dependent statistics at inference, the lecture derives the exponential-moving-average update for running mean and variance, which is used instead of batch statistics at test time.

Regularization and overparameterization (34:02)

The lecture explains that deep networks are typically overparameterized — they have more weights than training examples and can fit the training set almost exactly — which classically risks overfitting. It distinguishes implicit regularization (for example, SGD with a given initialization only reaches a limited set of functions) from explicit regularization, which the rest of the lecture focuses on.

L2 regularization / weight decay (40:17)

L2 regularization adds a penalty on the squared norm of the weights to the loss, on the reasoning that smaller weights produce smoother, less complex functions. Working through the gradient of this augmented objective shows that the resulting update multiplies the old weights by a factor slightly less than 1 before each gradient step, which is why it is called weight decay. The lecture notes this is normally implemented inside the optimizer (applied to the gradient) rather than as a separate loss term, and cautions that its real effect on function complexity in deep networks is uncertain.

Dropout (55:38)

Dropout randomly zeroes a fraction p of each layer's activations during training and rescales the remaining ones by 1/(1-p) to preserve their expected scale; it is applied only during training, not at test time. Rather than framing dropout as forcing robustness to missing features, the lecture presents it as a stochastic approximation of a layer's exact computation, directly analogous to how mini-batch SGD approximates full-batch gradient descent.

Interaction of the techniques and open questions about batch norm (1:04:16)

The lecture reviews the many design choices in training a deep network (optimizer, initialization, normalization, regularization) and warns against exhaustive grid search. It then traces the history of batch norm as a case study: the original paper attributed its benefit to reducing "internal covariate shift," a later paper argued instead that it smooths the optimization landscape, and still later work found no evidence that batch norm affects smoothness at all. It closes by noting that batch norm has since found new use as a way to make classifiers more robust under distribution shift, illustrating that even standard, heavily used techniques are not fully understood theoretically.

Before you watch

• Be comfortable with how deep networks compute forward and backward passes, and how gradient descent updates weights, from earlier lectures in this course.
• Know how weight initialization variance is chosen for ReLU networks, covered in an earlier lecture.
• Basic familiarity with mean, variance and the Frobenius norm of a matrix will help with the normalization and L2 regularization derivations.

Check your understanding

1. Why does a weight initialization of 1/n cause training to stall, while 3/n causes the loss to become NaN, for a deep ReLU network?
2. What is the difference between what layer norm normalizes and what batch norm normalizes, and why does that difference matter at test time?
3. Derive, in words, why the L2-regularized gradient update multiplies the weights by a factor less than 1 at each step.
4. Why is dropout described in this lecture as a stochastic approximation rather than as forcing robustness to missing activations?
5. What competing explanations does the lecture give for why batch norm improves training, and what does their disagreement suggest about our understanding of deep learning tricks?