Lecture 6: Fully Connected Networks, Optimization, Initialization

What this lecture covers

With automatic differentiation in place, the lecture revisits fully connected networks (multilayer perceptrons) using the simpler, bias-inclusive definition that autograd now makes practical, then moves into two topics that determine whether a network trains well at all: how to update parameters (optimization) and how to set their starting values (initialization). It covers matrix broadcasting subtleties in implementing bias addition, several optimization update rules used in practice, and why naive weight initialization breaks deep networks.

After watching, you can write the matrix form of an L-layer MLP including bias, explain the tradeoffs between gradient descent, Newton's method, momentum, Nesterov momentum, and Adam, and explain why random (not zero) initialization with a specific variance is needed for deep ReLU networks.

Key ideas

• MLP with bias: an L-layer network defines z_{i+1} = sigma_i(z_i W_i + b_i), with the bias term now included because automatic differentiation removes the burden of hand-deriving its gradient.
• Broadcasting for bias addition: adding a bias vector to a batch of activations requires broadcasting it across rows rather than explicitly forming a matrix of repeated copies, since explicit copies waste memory; needle implements broadcasting explicitly via reshape then broadcast_to.
• Gradient descent bounces on ill-conditioned functions: on a quadratic objective with different curvature in different directions, gradient descent zig-zags, and the step size trades off convergence speed against how much the parameters oscillate.
• Newton's method: scales the gradient by the inverse Hessian and solves quadratic objectives exactly in one step, but is impractical for deep learning because the Hessian is a parameter-count-squared matrix that cannot be formed or inverted at scale.
• Momentum: maintains an exponential moving average u of past gradients and steps using u instead of the raw gradient, smoothing out oscillation; bias correction (dividing by 1 - beta^(t+1)) compensates for the momentum term starting near zero.
• Nesterov momentum: evaluates the gradient at a look-ahead point based on the current momentum term rather than at the current iterate, giving smoother, sometimes provably faster convergence on convex problems.
• Adam: combines a first-moment momentum term with a second-moment (element-wise squared gradient) term, dividing the update by the square root of the second moment so all parameters are updated on a similar scale; it is one of the two optimizers (with plain SGD-plus-momentum) the lecture says are essential to know.
• Stochastic updates: in practice every method above is applied to a mini-batch gradient rather than the full-batch gradient, trading some noise for far cheaper iterations.
• Zero initialization fails: initializing all weights (and biases) to zero makes every gradient beyond the last layer zero as well, so a multi-layer network never leaves that fixed point.
• Initialization variance matters: random Gaussian initialization works only with the right variance; too large or too small a variance causes activation norms to explode or vanish across 50 layers, and ReLU networks need variance 2/n (Kaiming/He initialization) to account for roughly half the activations being zeroed by ReLU.

Walkthrough

Fully connected networks with bias and matrix broadcasting (1:01)

The lecture redefines the MLP with an explicit bias term per layer, made practical now that autodiff handles gradients automatically. It then works through why adding a bias vector to a batch matrix requires broadcasting rather than literally forming a matrix of copies (1 * b^T), which would waste memory, and shows the reshape-then-broadcast_to pattern needle uses, noting that implementing the adjoint of a broadcast operation for the homework is not entirely trivial.

Gradient descent recap and a quadratic test case (15:18)

Optimization is framed generically as minimizing some function f(theta). Gradient descent is restated with explicit iteration indices, then visualized on a simple two-dimensional quadratic objective f(theta) = theta^T P theta + q^T theta, showing how a larger step size converges faster but oscillates more in parameter space, while a smaller step size converges more smoothly but slowly.

Newton's method and why it doesn't scale to deep learning (23:41)

Newton's method scales the gradient by the inverse Hessian and, for the quadratic test function, reaches the optimum in a single full step because the quadratic approximation used by Newton's method is exact for quadratics. The lecture derives this exactness algebraically, then explains why the method is impractical for deep networks: the Hessian is an n-by-n matrix where n is the parameter count, which for models with millions to trillions of parameters cannot be formed, inverted, or even multiplied by, and its value for non-convex loss landscapes is unclear anyway.

Momentum and bias correction (37:25)

Momentum is introduced as a first-order alternative that keeps gradient descent's cost while damping oscillation. The update u_{t+1} = beta * u_t + (1-beta) * grad is expanded to show it forms a geometric (exponentially weighted) sum of past gradients, and the lecture walks through the visual effect on the quadratic test case for beta = 0.7 versus beta = 0. Because momentum starts small when u_0 = 0, a bias-correction factor of 1 - beta^(t+1) is introduced to rescale early updates back to gradient-sized steps.

Nesterov momentum (49:05)

Nesterov momentum is presented as a variant that evaluates the gradient at a point shifted by the current momentum term before forming the update, rather than at the current parameter value. The lecture notes this has stronger convergence guarantees for convex problems and tends to produce a smoother trajectory in practice, without claiming the same guarantees carry over to non-convex deep learning.

Adam (52:15)

Adam is described as one of the most durable optimization methods in deep learning, alongside plain SGD with momentum. It tracks two momentum-style terms: u (a moving average of the gradient) and v (a moving average of the element-wise squared gradient), then updates parameters using u divided by sqrt(v) + epsilon, all computed element-wise. This rescales each parameter's update by its own gradient magnitude, addressing the problem that different parameters and directions can have very different gradient scales. Bias correction, dividing each term by 1 - beta^(t+1) for its respective beta, is standard in Adam implementations.

Stochastic (mini-batch) variants (1:03:46)

Because the real training objective is an average of per-example losses, every method above is in practice applied to a gradient computed on a mini-batch rather than the full dataset. The lecture emphasizes that many cheap, noisy steps are often far more efficient than fewer exact steps, and that the plots shown for the quadratic test function only go so far in building intuition since real deep networks are non-convex.

Why zero initialization fails (1:10:04)

Unlike in convex optimization, where initializing parameters to zero is often fine, initializing an MLP's weights (and biases) to zero makes every layer's output zero, and consequently every gradient beyond the last layer zero as well: the all-zero point is a fixed point (a bad saddle point) that gradient descent can never leave.

Random initialization and the right variance (1:13:25)

Sampling weights from a Gaussian with mean zero is shown, using a 50-layer MNIST network with ReLU activations, to behave very differently depending on the chosen variance: 2/n keeps activation norms and gradient magnitude roughly constant across layers, 3/n causes them to blow up, and 1/n causes them to shrink toward zero. An informal central-limit-theorem argument explains why variance 1/n preserves variance through a purely linear layer, and why ReLU's zeroing of roughly half the activations means variance 2/n (Kaiming/He initialization) is needed to preserve variance through a ReLU network.

Before you watch

• Review the earlier lecture that hand-derived backpropagation for fully connected networks, since this lecture assumes that background and simplifies it using automatic differentiation.
• Complete or be familiar with the automatic differentiation homework, since broadcasting and gradient mechanics referenced here connect directly to it.
• Be comfortable with basic matrix calculus (gradients, and ideally what a Hessian is) and with random variables, means, and variances, since the initialization discussion uses the central limit theorem informally.

Check your understanding

1. Why does adding a bias vector to a batch of activations require broadcasting rather than a plain matrix addition, and why is explicitly forming the broadcasted matrix wasteful?
2. Why does Newton's method solve a quadratic optimization problem exactly in one step, and why is this approach impractical for deep networks?
3. What problem does momentum's bias-correction term solve, and why does it matter more in early iterations?
4. How do Adam's two moving-average terms differ in purpose, and how are they combined to form a parameter update?
5. Why does initializing all weights to zero prevent a multi-layer network from learning anything?
6. Why does a ReLU network need weight variance 2/n rather than 1/n to keep activation magnitudes stable across many layers?