Lecture 6: Singular Value Decomposition (SVD)

What this lecture covers

Since a rectangular matrix has no eigenvalues in the usual sense, Strang introduces a replacement that works for every matrix: the singular value decomposition, A = UΣV^T. He builds it from A^T A, a symmetric positive definite matrix whose eigenvectors become the right singular vectors V, while the corresponding U = AV/σ vectors turn out to be the eigenvectors of AA^T and, crucially, orthogonal to each other.

The lecture also gives the geometric picture of the SVD as rotate, stretch, rotate, counts how many parameters each factor contributes, and derives the polar decomposition as a quick corollary. It ends by identifying the leading rank-one piece σ1 u1 v1^T as the most important part of a data matrix, setting up the next lecture on principal components. After watching you should be able to state what the SVD factors are, explain why the singular vectors are orthogonal, and describe the rotate-stretch-rotate geometric picture.

Key ideas

• Why eigenvalues fail for rectangular matrices: Ax = λx requires A to map a space to itself, which is impossible when A is not square, so the SVD uses two different sets of vectors instead.
• A^T A is the key matrix: it is always square, symmetric, and positive semidefinite, so it factors as VΛV^T, giving the right singular vectors V as its eigenvectors and the singular values σ as the square roots of its eigenvalues.
• AA^T gives the left singular vectors: this matrix (a different size than A^T A) is also symmetric positive semidefinite, and its eigenvectors are the U vectors; the two matrices share the same nonzero eigenvalues.
• Orthogonality of AV: choosing U_i = AV_i / σ_i and using that the V_i are orthogonal eigenvectors of A^T A proves the resulting U_i vectors are also orthogonal, which is the key fact making the SVD work.
• Rotate, stretch, rotate: A = UΣV^T means multiplying by A first rotates (via V^T), then stretches along axes by the singular values (via Σ), then rotates again (via U).
• Parameter counting: for a 2x2 matrix, 4 entries split into 1 rotation angle (V), 2 stretch factors (Σ), and 1 rotation angle (U), matching the 4 degrees of freedom of the original matrix.
• Polar decomposition: inserting U^T U = I into A = UΣV^T gives A = (UΣU^T)(UV^T), a symmetric matrix times an orthogonal matrix, the matrix analog of writing a complex number as a magnitude times e^(iθ).
• Leading rank-one piece: σ1 u1 v1^T, built from the largest singular value, is the single most important rank-one piece of a data matrix, the starting point for principal component analysis.

Walkthrough

Why rectangular matrices need singular vectors, not eigenvectors (0:01)

Strang explains that Ax = λx cannot hold for a rectangular matrix since Ax lands in a different-dimensional space than x. The SVD replaces one set of eigenvectors with two: left singular vectors U and right singular vectors V, connected by singular values σ on a diagonal matrix in between, generalizing the spectral theorem to every matrix.

Building V and sigma from A^T A (4:04)

Strang shows A^T A is always symmetric and positive semidefinite, so it factors as VΛV^T, giving the eigenvectors V and eigenvalues λ = σ^2. Separately, AA^T, a different-sized matrix with the same nonzero eigenvalues, gives the eigenvectors U. Substituting A = UΣV^T into A^T A and using U^T U = I confirms this matches the eigendecomposition of A^T A exactly.

Proving the U vectors are orthogonal (17:12)

To handle repeated eigenvalues correctly, Strang defines U_i = A V_i / σ_i directly rather than relying only on AA^T. He then proves U1^T U2 = 0 by substituting the definitions, using that V2 is an eigenvector of A^T A with eigenvalue σ2^2, and that V1 and V2 are orthonormal, so the singular vectors in the column space inherit orthogonality from the singular vectors in the row space. He also notes that computing the SVD in practice avoids forming A^T A directly, since doing so squares the matrix's sensitivity to rounding error.

The rotate-stretch-rotate picture and parameter counts (28:23)

Strang draws the geometric picture: multiplying a unit circle by V^T rotates it, multiplying by Σ stretches it into an ellipse along the singular value directions, and multiplying by U rotates the ellipse again. He counts parameters for a 2x2 matrix (1 rotation angle from V, 2 stretch factors from Σ, 1 rotation angle from U, totaling 4, matching the matrix's 4 entries) and notes that U = V exactly when the original matrix is symmetric positive definite, in which case the SVD reduces to the spectral theorem.

Polar decomposition and the leading rank-one piece (45:39)

Inserting U^T U = I into A = UΣV^T gives A = (UΣU^T)(UV^T), the polar decomposition of A into a symmetric matrix times an orthogonal matrix, paralleling the magnitude-times-rotation form of a complex number. Strang closes by identifying σ1 u1 v1^T, built from the largest singular value, as the most important rank-one piece of a data matrix, the starting point for finding principal components in the next lecture.

Before you watch

• Watch Lecture 4 on eigenvalues and symmetric matrices and Lecture 5 on positive definite matrices first, since the SVD is built directly from the fact that A^T A is symmetric positive semidefinite.
• Comfort with the spectral theorem S = QΛQ^T is assumed throughout.

Check your understanding

1. Why can't a rectangular matrix have eigenvalues in the usual sense, and what replaces them in the SVD?
2. How do the eigenvectors of A^T A become the right singular vectors V, and where do the singular values come from?
3. What is the key step in proving that the vectors U_i = A V_i / σ_i are orthogonal to each other?
4. What does each of the three factors U, Σ, and V^T do geometrically to a unit circle?
5. How does the polar decomposition A = SQ follow directly from the SVD?