Lecture 18: Counting Parameters in SVD, LU, QR, Saddle Points

What this lecture covers

The lecture opens with a review exercise that ties together the course's factorizations: for each major decomposition of a matrix - LU, QR, eigenvector form, symmetric form, polar decomposition, and SVD - Strang counts the free parameters in the pieces and checks that they add up to the number of free parameters in the original matrix. This builds a concrete sense of how much information each factorization actually carries, including the harder case of a rank-R matrix. The lecture then turns to a new topic, saddle points, starting with the source that comes from constrained optimization.

This sits near the end of the course's linear algebra portion, just before the material shifts to probability and then optimization for deep learning. After watching, you should be able to count free parameters in any of the standard factorizations and explain why a constrained quadratic minimization problem produces a saddle point rather than a minimum.

Key ideas

• Free parameter counting: each factorization of an N x N matrix (N^2 parameters) should split its parameters across its factors so the totals add up to N^2, given the constraints each factor carries (unit length, orthogonality, triangularity).
• Orthogonal matrix Q: has 1/2 N(N-1) free parameters, since each successive column loses one degree of freedom to normalization and one more to each orthogonality condition with earlier columns.
• Symmetric matrix S: has 1/2 N(N+1) free parameters (the upper triangle plus the diagonal), matching the count from its eigendecomposition Q Lambda Q^T.
• Rank-R matrix parameter count: a rank-R matrix among all M x N matrices has R(M + N - R) free parameters, found by counting the reduced SVD's U, Sigma, V pieces.
• Rank-R matrices form a manifold, not a subspace: adding two rank-2 matrices generally gives a matrix of higher rank, so the set of fixed-rank matrices is a curved surface in matrix space rather than a vector space.
• Lagrangian and saddle points: minimizing a positive-definite quadratic x^T S x subject to linear constraints Ax = B leads to the Lagrangian L(x, lambda), whose stationary point is a saddle point, not a minimum, because of the added Lagrange multipliers.
• KKT matrix sign pattern: the block matrix [[S, A^T], [A, 0]] from this Lagrangian has both positive and negative eigenvalues; block elimination shows the first block of pivots (from S) is positive and the second block (from -A S^{-1} A^T) is negative, and pivot signs match eigenvalue signs.

Walkthrough

Counting parameters in LU, QR, and eigenvector factorizations (1:03)

Strang walks through each factorization in turn: L and U split N^2 parameters as 1/2 N(N-1) and 1/2 N(N+1); the eigenvector matrix X contributes N^2 - N parameters (after fixing a scaling convention) plus N eigenvalues; and the orthogonal matrix Q contributes 1/2 N(N-1) parameters because each column loses one degree of freedom to normalization and more to orthogonality with prior columns. Each case is checked to sum back to N^2.

Symmetric matrices and the polar decomposition (7:15)

The count for a symmetric matrix S is 1/2 N(N+1), matching the parameters in its eigendecomposition Q Lambda Q^T. Strang then applies the same reasoning to the polar decomposition A = Q S, noting its close relationship to the SVD.

Counting parameters in the SVD (10:18)

For a full-rank M x N matrix (M <= N), Strang counts parameters in U, Sigma, and V^T, carefully noting that only the first M columns of V matter since the rest come from the null space. The total works out to MN, matching the original matrix.

Parameters in a rank-R matrix (19:39)

Generalizing to a matrix of rank R less than full rank, the same style of count through the reduced SVD gives R(M + N - R) free parameters. Strang notes that fixed-rank matrices do not form a subspace: summing two rank-R matrices generally increases the rank, so the set of rank-R matrices is a lower-dimensional surface, not a vector space.

Saddle points from constraints (26:51)

Turning to saddle points, Strang sets up the constrained minimization of a positive-definite quadratic x^T S x subject to Ax = B, and introduces the Lagrangian L(x, lambda) = x^T S x + lambda^T(Ax - B). Taking derivatives with respect to x and lambda gives the block system Sx + A^T lambda = 0 and Ax = B.

The KKT matrix and its eigenvalue signs (32:03)

The resulting block matrix, of the form [[S, A^T], [A, 0]], is shown to be indefinite: the zero block on the diagonal rules out positive definiteness. Using a small 2x2 example, Strang demonstrates the eigenvalues split in sign, then uses block elimination on the general matrix to show the first set of pivots (from S) are positive and the second set (from -A S^{-1} A^T) are negative, tying pivot signs to eigenvalue signs for symmetric matrices.

Before you watch

• Review the SVD, LU, and QR factorizations from earlier lectures, since this lecture assumes familiarity with all of them.
• Recall the eigendecomposition of a symmetric matrix and the meaning of positive definiteness.

Check your understanding

1. Why does an N x N orthogonal matrix have 1/2 N(N-1) free parameters rather than N^2?
2. How is the free-parameter count for a rank-R matrix different from the count for a full-rank SVD, and why?
3. Why is a saddle point, rather than a minimum, the natural stationary point of a Lagrangian for constrained minimization?
4. What does the sign pattern of the pivots in the KKT matrix tell you about its eigenvalues?