Lecture 10: Survey of Difficulties with Ax = b

What this lecture covers

Strang steps back from any single algorithm and asks a broader question: given Ax = b, what actually goes wrong in practice, and what do you do about it? He works through a list of cases ranging from the easy, well-conditioned square system to underdetermined deep-learning-style systems, nearly singular matrices, and matrices too large to even form A^T A. The lecture is a map of the rest of the chapter rather than a deep derivation of any one method.

This sits early in the course's numerical linear algebra chapter, right after the pseudoinverse was introduced. After watching, you should be able to name which regime a given Ax = b problem falls into and know roughly which tool - elimination, least squares, Gram-Schmidt, regularization, iteration, or randomized sampling - applies to it. The lecture also works out, in full detail, why adding a penalty term to least squares recovers the pseudoinverse as the penalty shrinks to zero.

Key ideas

• Condition number: the ratio of the largest to smallest singular value, sigma_1 / sigma_n; a "good" square matrix keeps this under roughly a thousand and ordinary elimination (MATLAB's backslash) works fine.
• Underdetermined systems: when there are more unknowns than equations, as in deep learning's weight matrices, there are infinitely many solutions and an algorithm (like gradient descent) must pick one - the question of which one it picks is called implicit bias.
• Gram-Schmidt: replaces a matrix's columns with an orthonormal basis, connected to the original columns by a triangular matrix; needed when columns are nearly dependent, since inverting such a matrix directly is numerically dangerous.
• Inverse problems: recovering an unknown system (such as a circuit's components) from observed input-output behavior; these are typically nearly singular because many different systems produce almost the same output.
• Regularization (ridge regression): minimizing ||Ax - b||^2 + delta^2 ||x||^2 instead of plain least squares keeps a nearly singular problem solvable by adding a small penalty on the size of x.
• Pseudoinverse as a limit: as the penalty delta goes to zero, the regularized solution converges to the pseudoinverse solution, with a sharp split between the nonzero-singular-value case (converges to 1/sigma) and the zero-singular-value case (converges to 0).
• Randomized linear algebra: when a matrix is too large to even form A^T A, sampling its rows and columns gives a good approximate answer with high probability.

Walkthrough

Cataloguing the cases of Ax = b (0:01)

Strang opens by listing the situations Ax = b can present: a well-conditioned square system, too many equations (least squares), too few equations (underdetermined, as in deep learning), columns that are nearly dependent, matrices that are nearly singular, matrices too big to form A^T A, and matrices so big they don't even fit in memory. He frames the lecture as a practical dictionary - what symptom tells you which tool to reach for.

The good normal case and its near neighbors (2:03)

A reasonably sized, reasonably conditioned square matrix is solved directly by elimination (MATLAB's backslash). Too many equations leads to least squares and the normal equations, still solvable directly if A^T A is a manageable size. Too few equations is the underdetermined case central to deep learning: many solutions exist, and you either pick the minimum-norm (L2) solution or ask whether an algorithm like stochastic gradient descent implicitly favors a minimum-L1 solution - an open question Strang flags as unresolved at the time.

Bad column conditioning and Gram-Schmidt (6:05)

When a matrix's columns are nearly linearly dependent, its inverse becomes very large even though it technically exists. The fix is to orthogonalize the columns with Gram-Schmidt, producing an orthonormal basis Q connected to the original columns A by a triangular matrix R. Strang previews two orderings of Gram-Schmidt - the standard textbook version and a better-organized version that supports column pivoting, reordering columns the way elimination reorders rows when a pivot is too small - but defers the full treatment to the next lecture.

Inverse problems and why they're nearly singular (12:07)

An inverse problem asks you to recover a system, such as an electrical network's resistances and inductances, from observed responses to known inputs. These problems are typically nearly singular because many different underlying networks produce almost identical outputs. Strang introduces the standard fix: add a penalty term to least squares so the problem stays solvable, which raises the practical question - not answered here - of how large that penalty should be.

Big and very big matrices: iteration and randomization (16:13)

For large but still workable matrices, iterative methods such as conjugate gradients get progressively closer to an answer without ever computing it exactly. For matrices too large to even look at directly - where forming A^T A is unthinkable - randomized linear algebra samples the columns and rows of A and estimates the answer from the sample, using probability bounds to argue the estimate is very likely close to correct.

Working out the pseudoinverse as a limit (24:21)

Strang sets up the penalized least squares problem min ||Ax - b||^2 + delta^2 ||x||^2, shows it is equivalent to solving (A^T A + delta^2 I) x = A^T b, and then works the case of a 1x1 matrix (just a number sigma) by hand. Minimizing (sigma x - b)^2 + delta^2 x^2 and setting the derivative to zero gives x = sigma b / (sigma^2 + delta^2). As delta goes to zero, this approaches 1/sigma when sigma is nonzero, but stays at 0 when sigma is exactly zero - the same sharp split that defines the pseudoinverse. He states, without full proof, that the same limit holds for any matrix A, provable by substituting the SVD A = U sigma V^T, which reduces the general case to the diagonal case and then to this 1x1 case on each diagonal entry.

Before you watch

• Know the SVD (A = U sigma V^T) and the pseudoinverse from the previous lecture in this course, since this lecture builds directly on both.
• Be comfortable with least squares and the normal equations A^T A x = A^T b.
• Understand what a condition number is and why a large one signals a numerically risky matrix.

Check your understanding

1. Why does a nearly singular matrix cause problems even though it is technically invertible?
2. In the 1x1 case, why does the penalized solution sigma b / (sigma^2 + delta^2) split into two different limits depending on whether sigma is zero or nonzero?
3. What distinguishes an inverse problem from an ordinary Ax = b problem, and why does that make it prone to being nearly singular?
4. Why does randomized linear algebra become necessary once a matrix is too large to form A^T A?
5. What is the underdetermined case in Ax = b, and why does Strang connect it to deep learning?