Lecture 8: Norms of Vectors and Matrices

What this lecture covers

This lecture is a systematic tour of norms, ways to measure the size of a vector, matrix, or function. Strang reviews the Lp family of vector norms (L1, L2, L-infinity, and the problematic L0), then builds geometric intuition by drawing the unit ball of each: a diamond, a circle, and a square in two dimensions. He uses this picture to explain why minimizing in the L1 norm tends to produce sparse solutions, a fact central to compressed sensing.

The lecture then extends these ideas to matrices, revisiting the spectral (L2), Frobenius, and nuclear norms from the previous lecture and showing how each connects to a vector norm of the singular values. It opens with an aside about a probability puzzle from a colleague's talk and closes with a research conjecture linking neural network training to the nuclear norm. After watching you should be able to sketch the unit ball for the L1, L2, and L-infinity norms, explain why L1 minimization favors sparse solutions, and state how each matrix norm relates to the singular values.

Key ideas

• The Lp family of vector norms: ||v||_p = (sum of |v_i|^p)^(1/p); L2 is ordinary length, L1 is the sum of absolute values, and L-infinity is the largest entry, with L0 (the count of nonzero entries) not being a true norm since it fails to scale with the vector.
• Unit ball shapes: the set of vectors with norm 1 is a diamond for L1, a circle for L2, and a square for L-infinity; these shapes are convex, while the L0 "unit ball" and any p < 1 case are not.
• L1 favors sparsity: minimizing the L1 norm subject to a linear constraint tends to land on a solution with more zero components than minimizing L2, because the diamond's corners touch the axes first as the norm grows from the origin.
• Weighted (S) norm: for a positive definite symmetric matrix S, sqrt(v^T S v) defines a valid norm whose unit ball is an ellipse; using the identity matrix for S recovers the ordinary L2 norm.
• Matrix 2-norm from vector norms: the matrix 2-norm is defined as the maximum ratio ||Ax|| / ||x|| over all vectors x, and this maximum equals the largest singular value σ1, achieved at x = v1, the first right singular vector.
• Frobenius and nuclear norms revisited: the Frobenius norm equals the square root of the sum of the squares of all singular values, and the nuclear norm equals the sum of the singular values; both are unchanged by multiplying by an orthogonal matrix, since orthogonal matrices preserve norms.
• Nuclear norm and deep learning: a research conjecture mentioned in the lecture proposes that gradient descent training of certain neural networks, which have far more weights than data samples, tends to select the solution that minimizes the nuclear norm among all solutions achieving the same loss.

Walkthrough

The Lp family and unit-ball geometry (4:06)

Strang reviews the L2, L1, and L-infinity vector norms, then explains why the L0 norm (counting nonzero entries) fails to be a true norm since doubling a vector doesn't double this count. Plotting the set of unit-norm vectors in 2D for each case produces a circle (L2), a diamond (L1), and a square (L-infinity), and Strang notes that as p decreases toward 1 these shapes remain convex, while going below p = 1 produces a non-convex shape that violates the triangle inequality.

Why L1 minimization gives sparse solutions (22:33)

Using a simple constrained minimization problem, find the point on a given line closest to the origin, Strang compares the L2 winner (the ordinary nearest point) to the L1 winner (found by growing the diamond from the origin until it first touches the line). The L1 solution lands exactly on an axis, meaning one of its components is zero, illustrating in two dimensions why L1 minimization favors sparse vectors more generally.

Weighted norms and the S norm (17:30)

Strang defines a norm built from a positive definite symmetric matrix S, sqrt(v^T S v), the same energy expression from earlier lectures, now used as a norm. Its unit ball is an ellipse whose axis lengths depend on the entries of S, and choosing S as the identity matrix recovers the ordinary L2 norm as a special case.

Matrix norms and the singular values (31:55)

Strang defines the matrix 2-norm as the maximum blow-up factor ||Ax|| / ||x|| over all vectors x, and shows this maximum is achieved at x = v1, the first right singular vector, giving ||Av1|| / ||v1|| = σ1. He then revisits the Frobenius norm, showing it equals the square root of the sum of the squares of the singular values (using that orthogonal matrices don't change the norm to simplify A = UΣV^T down to Σ alone), and the nuclear norm as the sum of the singular values, an L1-type norm applied to the vector of singular values.

Before you watch

• Watch Lecture 6 on the singular value decomposition and Lecture 7 on matrix norms and the Eckart-Young theorem first, since this lecture builds directly on the spectral, Frobenius, and nuclear norms introduced there.
• Familiarity with positive definite matrices and the energy function v^T S v from Lecture 5 is assumed for the weighted norm section.

Check your understanding

1. Why does the L0 "norm" fail to satisfy the scaling property required of a true norm?
2. What shape is the unit ball for the L1 norm in two dimensions, and why does that shape lead to sparse minimizers?
3. How is the matrix 2-norm defined in terms of vector norms, and why does it equal the largest singular value?
4. How does the Frobenius norm of a matrix relate to its singular values?
5. What does the S-norm sqrt(v^T S v) reduce to when S is the identity matrix, and what shape does its unit ball take for a general positive definite S?