Lecture 6: Training Neural Networks I

What this lecture covers

Lecture 5 introduced the convolutional layer as a building block; this lecture asks a different question: once you've designed a network, how do you actually get it to train well? It works through the practical choices that determine whether training succeeds at all, starting with which nonlinearity to use, then how to preprocess data and initialize weights so gradients neither vanish nor explode, then batch normalization as a way to keep activations well behaved throughout training, and finally how to babysit training in practice by sanity-checking the loss and searching for good hyperparameters.

After watching, you should be able to explain why sigmoid activations cause problems in deep networks, compute how zero-centering and normalization affect gradient updates, describe what goes wrong with naive weight initialization in deep networks and how Xavier initialization addresses it, explain what batch normalization computes and why it helps, and run the basic sanity checks and learning-rate search the lecture demonstrates. This is the first of two lectures on training mechanics; the next lecture continues with optimizers, regularization, and transfer learning.

Key ideas

• Saturating gradients: sigmoid and tanh flatten out for large positive or negative inputs, which drives the local gradient to zero and kills gradient flow through those units.
• Zero-centered outputs: if all outputs from a layer are positive, the gradients on the next layer's weights all share the same sign, forcing inefficient zig-zag updates instead of moving directly toward the optimum.
• ReLU: f(x) = max(0, x) avoids saturation on the positive side and is much cheaper to compute than sigmoid, but it isn't zero-centered and can produce "dead" units that never activate, often from a too-high learning rate.
• Data preprocessing: for images, the standard approach is to zero-center by subtracting a mean image or per-channel mean computed once on the training set, without further normalization or PCA/whitening.
• Weight initialization: initializing weights to all zeros or to numbers that are too small or too large causes activations to collapse to zero or saturate as they pass through a deep network.
• Xavier initialization: scales the random initial weights by the number of inputs so that the variance of activations stays roughly constant across layers; it needs an adjustment (dividing by 2) to work with ReLU.
• Batch normalization: normalizes each layer's activations to zero mean and unit variance using the statistics of the current mini-batch, then applies a learned scale (gamma) and shift (beta) so the network can recover other distributions if needed.
• Hyperparameter search: search learning rate and similar hyperparameters in log space, in coarse-then-fine stages, using random rather than grid sampling.

Walkthrough

Activation functions: sigmoid, tanh, ReLU (5:11)

The lecture reviews the sigmoid function and identifies three problems: saturated regions kill gradients (shown by walking through what happens to the local gradient at x = -10, 0, and 10), non-zero-centered outputs force all weight gradients in a layer to share a sign, producing slow zig-zag optimization, and the exponential is mildly expensive to compute. Tanh fixes the zero-centering problem but still saturates. ReLU avoids saturation on its positive half, is very cheap to compute, and converges roughly six times faster in practice, which is why it became standard after AlexNet, but it is still not zero-centered and can produce dead units that never fire, particularly when initialization is unlucky or the learning rate is too high.

ReLU variants (22:32)

The lecture covers several variants designed to fix ReLU's remaining problems: Leaky ReLU adds a small negative slope so units never fully die; parametric ReLU (PReLU) learns that negative slope as a parameter via backprop; the exponential linear unit (ELU) keeps a saturating negative region for added noise robustness while pushing outputs closer to zero mean; and Maxout takes the max of two separate linear functions, generalizing ReLU and Leaky ReLU at the cost of doubling the parameters per neuron. The practical takeaway given is to use ReLU by default, be careful with learning rates, and treat the other variants as experimental options worth trying.

Data preprocessing for images (27:40)

General machine learning practice zero-centers and normalizes data so all features contribute comparably, but for images, where pixel values already share a similar scale, the lecture recommends zero-centering only, without normalization or more elaborate preprocessing like PCA or whitening, since a ConvNet is meant to operate directly on spatial structure. The training-set mean (either a full mean image or a per-channel mean, as used in networks like VGG) is computed once and reused unchanged at test time.

Why weight initialization matters (34:49)

Initializing all weights to zero breaks nothing structurally but removes symmetry breaking: every neuron computes the same thing and gets the same gradient. Small random weights work for small networks, but the lecture shows an experiment with a 10-layer, 500-neuron tanh network where activations shrink toward zero at every layer and gradients vanish. Scaling weights up instead causes activations to saturate at -1 or 1 in every layer, again killing gradients. This demonstrates that initialization scale has a large effect on whether a deep network can train at all.

Xavier initialization (45:03)

Xavier initialization, from a 2010 paper by Glorot, scales the random initial weights by the number of inputs so the variance of a layer's output matches the variance of its input, keeping activations in a reasonable range through many layers. The derivation assumes linear activations near the origin, so it works with tanh but not directly with ReLU, which zeroes out roughly half of its inputs and effectively halves the output variance; the lecture shows that dividing the Xavier scale by 2 compensates for this and restores good activation distributions throughout a deep ReLU network.

Batch normalization (49:10)

Rather than only hoping initialization keeps activations well distributed, batch normalization enforces this directly: for each mini-batch, it computes the empirical mean and variance per feature (or per activation map, for convolutional layers) and normalizes activations to zero mean and unit variance, typically inserted after fully connected or convolutional layers. Because forcing a unit gaussian isn't always ideal, especially for nonlinearities like tanh, a learned scale gamma and shift beta are applied afterward, letting the network recover the original distribution, including the identity mapping, if that works better. Batch normalization improves gradient flow, makes training more robust to learning rate and initialization choices, and has a mild regularizing effect since each example's normalized output depends on the rest of its batch. At test time, running estimates of mean and variance from training are used instead of recomputing them.

Babysitting the learning process and hyperparameter search (1:05:31)

The lecture walks through a practical training checklist: preprocess the data, pick an architecture, check that the initial loss matches the expected value for a softmax classifier with small weights (around 2.3 for 10 classes), confirm that turning up regularization increases the loss, and verify the network can drive training loss to zero by overfitting a tiny subset of data. It then demonstrates searching for a learning rate: too small barely moves the loss even though accuracy can still jump; too large produces NaN losses from an exploding cost. Good learning rates for this setup were found roughly between 1e-3 and 1e-5. The lecture recommends coarse-to-fine cross-validation in log space, watching for hyperparameter values that land at the edge of the searched range (a sign the range should shift), and preferring random over grid search since it samples the most important hyperparameter more densely. It closes by describing how to read loss curves and the gap between training and validation accuracy to detect problems like a too-high learning rate, bad initialization, or overfitting.

Before you watch

• Review the convolutional layer, computational graphs, and backpropagation from earlier lectures, since this lecture builds directly on stacking Conv/linear layers with nonlinearities.
• Be comfortable with the chain rule and how gradients flow through a computational graph, since several explanations rely on tracing upstream and local gradients.
• Recall mini-batch stochastic gradient descent from earlier in the course, since the lecture assumes this training loop as a starting point.

Check your understanding

1. Why does a saturated sigmoid or tanh unit produce a near-zero gradient, and how does this affect training of deep networks?
2. Why does having all-positive inputs to a layer lead to inefficient, zig-zagging gradient updates on that layer's weights?
3. What goes wrong when a deep tanh network is initialized with weights that are too small, and separately, what goes wrong when they're too large?
4. How does Xavier initialization need to be adjusted for ReLU networks, and why?
5. What does batch normalization compute, and what role do the learned gamma and beta parameters play?
6. When searching for a good learning rate, what does it mean if the best values found are clustered at the edge of your search range, and what should you do about it?