Lecture 17: Rapidly Decreasing Singular Values

What this lecture covers

This guest lecture by Alex Townsend answers a question that sits behind much of the course's work on compression and data analysis: why do so many matrices that show up in practice turn out to be low rank, or close to it? Rather than proving a single theorem, the lecture builds intuition through examples, starting from flag images and moving to the Hilbert and Vandermonde matrices, before landing on a general explanation involving the Sylvester equation.

By the end you should be able to explain what makes a matrix low rank in the strict sense, distinguish exact rank from numerical rank, and describe two competing explanations - smoothness and the Sylvester equation - for why numerical low rank appears so often in computational mathematics.

Key ideas

• Low rank (strict definition): a matrix is low rank if it takes fewer numbers to send it as K singular triplets (u, sigma, v) than to send all N^2 entries directly, which holds once the rank K is well under N/2.
• Alignment with rows and columns: matrices whose structure lines up with the grid (like a flag with a single color block) tend to be very low rank; diagonal or triangular patterns are the opposite case and resist compression.
• Numerical rank: the rank of a matrix once you allow a tolerance epsilon; a matrix can be full rank in the exact sense while still being well approximated by a much lower-rank matrix.
• Hilbert matrix: a full-rank matrix whose singular values decay so fast that a 1000x1000 version can be approximated to 15 digits by a rank-28 matrix.
• Smoothness explanation (Reed): if a matrix comes from sampling a smooth function, that function can be approximated by a low-degree polynomial, which gives a bound on numerical rank - but for the Hilbert matrix this bound (around 719) is far too pessimistic.
• Sylvester equation: many low-numerical-rank matrices satisfy AX - XB = C for some matrices A, B, C of low rank; this reframes the compression question as a question about the eigenvalues of A and B.
• Zolotarev numbers: when the eigenvalue sets of A and B are well separated, a classical quantity called the Zolotarev number shrinks quickly with k, giving a much tighter bound on numerical rank (34 for the Hilbert matrix, versus 719 from the smoothness argument).

Walkthrough

Singular values and exact low rank (1:23)

The lecture opens by reviewing what singular values tell you about a matrix X: the count of nonzero singular values equals the rank, and a rank-K matrix can be written as a sum of K rank-one pieces u_i v_i^T. Townsend defines low rank precisely by comparing the cost of sending all N^2 entries of X against sending the K pairs of vectors needed for the low-rank form, which is efficient once K is meaningfully smaller than N/2.

What low-rank matrices look like (7:38)

Using a set of world flags as example matrices, the lecture shows that flags aligned with the grid, like Austria's single-color blocks, are rank one, while diagonal patterns like the Scottish flag are close to full rank. Working through the triangular ("half filled") matrix in detail, Townsend connects its inverse to a finite-difference matrix Gilbert Strang has studied extensively, and uses that connection to show its singular values are all large and do not decay - explaining why diagonal patterns compress badly.

The Japanese flag and circles (14:56)

The lecture then explains why a circle, which looks like the "anti-aligned" case, is still low rank. By decomposing the circle into a rank-one piece plus a small square-shaped remainder and using the symmetry of the circle to bound the rank of that remainder, Townsend derives a rank bound that depends on the circle's radius and turns out to be small.

Numerical rank (19:03)

Numerical rank is introduced as exact rank with a tolerance epsilon: the numerical rank is K if singular value K+1 and beyond fall below epsilon relative to the largest singular value. This matters in practice because computers already round to a fixed number of digits, so a matrix that is numerically low rank to 16-digit precision is effectively indistinguishable from an exact low-rank matrix on a computer.

Hilbert and Vandermonde matrices (24:16)

Two classic examples are introduced: the Hilbert matrix, which is full rank but numerically very low rank, and the Vandermonde matrix from polynomial interpolation, whose numerical low rank makes it notoriously hard to invert accurately.

The smoothness argument and its limits (27:17)

The lecture presents the traditional explanation, attributed to John Reed: matrices sampled from smooth functions are well approximated by low-degree polynomials, so they inherit a polynomial bound on rank. Applying this to the Hilbert matrix, Townsend shows the bound it produces (about 719) is far larger than the true numerical rank of 28, meaning the smoothness argument, while intuitive, is quantitatively weak for this case.

The Sylvester equation and Zolotarev numbers (36:34)

As a sharper alternative, the lecture introduces the Sylvester equation AX - XB = C and shows that both the Hilbert and Vandermonde matrices satisfy such an equation with low-rank C. A known bound relates the singular values of X to Zolotarev numbers, which depend on how separated the eigenvalues of A and B are. For the Hilbert matrix this approach gives a numerical rank bound of 34, far closer to the true value of 28 than the smoothness bound.

Before you watch

• Be comfortable with the SVD and the meaning of singular values, covered in earlier lectures of this course.
• Familiarity with the Hilbert matrix and finite-difference matrices is helpful but not required, since the lecture reviews the needed properties.

Check your understanding

1. Why does a triangular (half-filled) matrix compress poorly compared to a matrix aligned with the grid?
2. What is the difference between a matrix being exactly low rank and being numerically low rank?
3. Why does the smoothness-based bound on the Hilbert matrix's rank turn out to be far too loose?
4. What role do the eigenvalues of A and B play in bounding the numerical rank of a matrix that satisfies a Sylvester equation?