Lecture 7: Training Neural Networks II

What this lecture covers

This lecture continues directly from Lecture 6's training mechanics, moving from getting a network to train at all toward making training faster and generalization better. It first digs into why plain stochastic gradient descent struggles, then builds up a family of fancier optimizers, momentum, Nesterov momentum, AdaGrad, RMSProp, and Adam, that address those problems. It then turns to the gap between training and test performance: model ensembles as a cheap way to improve results, dropout and related noise-based regularization techniques for single models, and finally transfer learning as a way to get good performance with limited data by starting from a network pretrained on a large dataset like ImageNet.

After watching, you should be able to explain why SGD struggles with poorly conditioned loss landscapes, saddle points, and noisy gradients, describe how momentum, AdaGrad, RMSProp, and Adam each modify the basic gradient step, know when to use learning rate decay, and articulate how dropout, data augmentation, and transfer learning each help a network generalize better or train with less data.

Key ideas

• Poor conditioning: when a loss landscape is far more sensitive in one direction than another, plain SGD zig-zags and converges slowly; this problem grows worse in the very high-dimensional spaces of real neural networks.
• Saddle points: in high dimensions, points where the gradient is zero but the loss increases in some directions and decreases in others are far more common than true local minima, and SGD (or gradients near such points) can stall badly.
• Momentum: accumulates a decaying "velocity" from past gradients and steps in that direction, which helps carry the optimizer through saddle points, local dips, and noisy gradients, and dampens zig-zagging.
• AdaGrad and RMSProp: divide each parameter's update by a running estimate of its squared gradients, accelerating slow dimensions and damping fast ones; RMSProp decays that estimate over time so steps don't shrink to nothing the way AdaGrad's can.
• Adam: combines a momentum-like first-moment estimate with an RMSProp-like second-moment estimate, plus a bias correction for early training steps; a common default is beta1=0.9, beta2=0.999, learning rate around 1e-3 to 5e-4.
• Learning rate decay: reducing the learning rate over training (by steps or continuously) can help a model settle into a good region once progress stalls, though it's treated as a secondary hyperparameter tuned after the base learning rate.
• Dropout: randomly zeroing activations during each forward pass regularizes a network by preventing co-adaptation of features and approximates training a large ensemble of shared-weight subnetworks; at test time, outputs are scaled instead of applying randomness.
• Transfer learning: reusing a network pretrained on a large dataset (typically ImageNet) by reinitializing and training only the last layer for a small dataset, or fine-tuning more layers when more data is available.

Walkthrough

Optimization landscape problems with SGD (16:19)

The lecture illustrates why vanilla SGD, which just steps in the negative gradient direction, runs into trouble. When the loss is far more sensitive in one direction than another (a "taco shell" shape, described by a poor condition number), SGD zig-zags rather than heading directly to the minimum, and this gets worse as dimensionality grows. At true local minima or saddle points, the gradient is zero, and SGD simply stops; the lecture argues that in high-dimensional networks with millions of parameters, saddle points, not local minima, are the dominant problem, since a local minimum requires every one of those millions of directions to increase loss. Because gradients are estimated from mini-batches rather than the full dataset, they are also noisy, which makes plain SGD meander. Switching to full-batch gradient descent does not fix any of these issues.

Momentum and Nesterov momentum (23:22)

Adding a momentum term, a velocity that decays by a friction hyperparameter (commonly 0.9) and accumulates gradients, lets the optimizer carry through saddle points and shallow local dips the way a ball rolling downhill keeps moving through flat spots, and it smooths out noisy gradient estimates. Momentum also helps with poor conditioning: zig-zagging components partially cancel while the consistent direction accelerates. Nesterov accelerated gradient is a variant that evaluates the gradient at a point shifted by the current velocity before combining it with that velocity, giving a kind of error-correction that slightly reduces overshoot compared to plain momentum.

AdaGrad, RMSProp, and Adam (32:40)

AdaGrad divides each update by the running sum of squared gradients seen so far for that parameter, which speeds up progress on consistently low-gradient dimensions and slows it on high-gradient ones, but its accumulated sum only grows, so steps shrink toward zero over long training runs and it can stall near saddle points. RMSProp fixes this by letting the squared-gradient estimate decay over time instead of accumulating forever. Adam combines a momentum-like moving average of gradients (first moment) with an RMSProp-like moving average of squared gradients (second moment), plus a bias correction that prevents oversized steps in the first few iterations when the second-moment estimate is still near zero. The lecture recommends Adam with beta1=0.9, beta2=0.999, and a learning rate near 1e-3 or 5e-4 as a strong default for most problems, though it notes none of these methods fix poor conditioning that isn't aligned with the parameter axes.

Learning rate decay and second-order optimization (44:54)

Rather than fixing one learning rate for all of training, it's common to decay it over time, by discrete steps or continuously, once progress plateaus; this pattern shows up as the characteristic step-drops in many published training loss curves. Learning rate decay is described as a secondary hyperparameter, tuned after finding a good base learning rate, and is more commonly paired with SGD plus momentum than with Adam. The lecture also introduces second-order optimization, which uses curvature (the Hessian) to fit a quadratic approximation to the loss and step directly toward its minimum without needing a learning rate; in principle this converges faster, but computing and inverting the full Hessian is infeasible for networks with millions of parameters, so exact second-order methods like L-BFGS are mostly used only in low-stochasticity settings such as style transfer rather than for standard network training.

Model ensembles (50:58)

All the optimizers discussed reduce training error, but the real goal is generalization. Training several independent models and averaging their predictions at test time, a model ensemble, reliably improves performance by a modest amount and is common in competitions like ImageNet. Cheaper variants include keeping multiple snapshots of one model during training (sometimes paired with a cyclical learning rate schedule to reach different good regions) or maintaining an exponentially decaying average of the parameters themselves (Polyak averaging).

Dropout and other regularization strategies (55:01)

To improve single-model generalization, dropout randomly zeroes a subset of activations at each layer on every forward pass, which can be understood as discouraging co-adaptation between features or as implicitly training a huge ensemble of subnetworks that share weights. At test time the randomness is removed and outputs are instead scaled by the keep probability to match the expected training-time activation (or, with inverted dropout, the scaling is moved to training time instead). The lecture frames dropout as one instance of a general pattern, adding stochasticity during training and averaging it out at test time, and shows that batch normalization and data augmentation (random crops, flips, and color jitter) fit the same pattern; more experimental variants mentioned include DropConnect, fractional max pooling, and stochastic depth.

Transfer learning (1:09:10)

Transfer learning addresses overfitting caused by too little data: start from a network pretrained on a large dataset such as ImageNet, reinitialize the final classification layer for your smaller number of classes, and train only that layer while freezing the rest. With more available data, larger portions of the network can be fine-tuned, usually with a reduced learning rate so the pretrained weights aren't disrupted too much. The lecture frames this as the norm rather than the exception in computer vision practice, since nearly every system, from detection to captioning, builds on a pretrained CNN, which is why deep learning frameworks ship "model zoos" of pretrained networks.

Before you watch

• Watch Lecture 6 first, since this lecture assumes familiarity with activation functions, weight initialization, batch normalization, and basic SGD from that session.
• Be comfortable with gradients and the idea of a loss landscape, since the optimizer comparisons rely on visualizing loss surfaces.
• Recall what overfitting and the train/validation gap mean, since regularization and transfer learning are both framed as ways to address it.

Check your understanding

1. Why do saddle points become more of a problem than local minima as the number of parameters grows very large?
2. How does adding a momentum term help an optimizer get past saddle points and noisy gradients?
3. What is the key difference between AdaGrad and RMSProp, and why does it matter for long training runs?
4. Why does Adam need a bias-correction step in its first few updates?
5. Why is model ensembling generally more expensive at test time than at training time, and what cheaper alternative achieves a similar effect?
6. How does dropout at training time relate to what happens at test time, and why is it acceptable to remove the randomness then?
7. When would you choose to fine-tune more layers of a pretrained network rather than just retraining the final layer?