Lecture 21: Minimizing a Function Step by Step

What this lecture covers

This lecture opens the course's optimization unit, the algorithmic foundation for deep learning. It develops the multivariable Taylor series through the gradient and Hessian, sets up Newton's method for solving systems of equations and for minimizing functions, and works a scalar example to show Newton's method converges quadratically. The second half introduces convexity, first for sets and then for functions, as the property that makes optimization problems well behaved.

This is the first of two lectures on optimization before the course moves to gradient descent specifically. After watching, you should be able to write the gradient and Hessian of a multivariable function, explain why Newton's method converges quickly but is expensive for large problems, and test whether a set or function is convex.

Key ideas

• Gradient and Hessian: for a function of n variables, the gradient is the vector of first partial derivatives and the Hessian H is the symmetric matrix of second partial derivatives; both appear in the three-term Taylor series f(x + Delta x) ~ f(x) + gradient^T Delta x + 1/2 Delta x^T H Delta x.
• Jacobian: for a vector-valued function F of n variables, the Jacobian is the matrix of first derivatives J_jk = dF_j/dx_k, generalizing the gradient to systems of equations.
• Newton's method for equations: iterates x_{k+1} = x_k - J^{-1} F(x_k), and for the scalar example f(x) = x^2 - 9, the error satisfies error_{k+1} = error_k^2 / (2 x_k), showing quadratic convergence near a root.
• Newton's method for minimization: uses the Hessian instead of the Jacobian, x_{k+1} = x_k - H^{-1} gradient(x_k), converging quadratically but requiring O(n^2) second-derivative entries per step.
• Steepest descent: moves in the direction of -gradient(f) with a chosen step size (learning rate); an exact line search picks the step that minimizes f along that direction, though in practice a fixed or backtracking step size is often used instead.
• Convex sets: a set is convex if the line segment between any two points in the set stays in the set; intersections of convex sets are always convex, but unions generally are not.
• Convex functions: a function is convex if the region on and above its graph (the epigraph) is a convex set; for a scalar function this matches the second-derivative test, and in n variables it corresponds to a positive semidefinite (or definite, for strict convexity) Hessian.

Walkthrough

The multivariable Taylor series (1:06)

Strang extends the familiar Taylor series to n variables, introducing the gradient as the vector of first derivatives and the Hessian as the symmetric matrix of second derivatives, noting that computing the full Hessian costs roughly n^2/2 entries, which becomes expensive for the very large problems deep learning involves.

Newton's method for systems of equations (10:31)

For solving F(x) = 0, Strang derives Newton's iteration using the Jacobian, then works the scalar example f(x) = x^2 - 9, showing algebraically that the error at each step is the square of the previous error divided by 2x_k - the source of Newton's method's fast, quadratic convergence near a solution.

Newton's method for optimization (22:48)

The same reasoning is applied to minimizing a function, replacing the Jacobian with the Hessian: x_{k+1} = x_k - H^{-1} gradient(x_k). Strang contrasts this with steepest descent, which uses only the gradient direction and a chosen step size, giving linear rather than quadratic convergence but avoiding the cost of computing the Hessian.

Convex sets (32:07)

Convexity is introduced as the key property for well-behaved optimization problems. A convex set is one where the segment between any two of its points stays inside it; Strang checks this with triangle examples, showing intersections of convex sets are always convex while unions generally are not.

Convex functions (44:29)

A function is defined as convex when the region on and above its graph is a convex set. Strang shows that the minimum of two convex functions is generally not convex (it has a kink), while the maximum of any collection of convex functions is convex. The lecture closes by connecting convexity to the Hessian: positive semidefinite for convexity, positive definite for strict convexity, matching the ordinary second-derivative test in one variable.

Before you watch

• Review the SVD and symmetric matrix material from earlier in the course, since positive definiteness is used throughout.
• Familiarity with single-variable Taylor series and derivatives (basic calculus) is assumed.

Check your understanding

1. Why does Newton's method converge quadratically, while steepest descent typically converges only linearly?
2. What makes computing the full Hessian expensive for large-scale problems like deep learning?
3. Why is the intersection of two convex sets always convex, while the union usually is not?
4. How does the Hessian test for convexity in several variables generalize the second-derivative test from one variable?