Lecture 11: Minimizing ‖x‖ Subject to Ax = b

What this lecture covers

This lecture opens with a geometric recap of a question from an earlier class: given one linear constraint, which point on that line has the smallest norm, and how does the answer change between the L1, L2, and L infinity norms? Strang then moves to the main topic, Gram-Schmidt orthogonalization, showing how it turns a matrix of independent but non-orthogonal columns into an orthonormal basis, and how that process is captured in the A = QR factorization.

The lecture sits in the numerical linear algebra portion of MIT 18.065, alongside earlier lectures on solving Ax = b and least squares. After watching, you should be able to explain what the R in QR actually contains, carry out Gram-Schmidt by hand including the column-pivoting variant used for numerical stability, and describe why Krylov spaces need to be orthogonalized before they are useful for large sparse problems.

Key ideas

• Minimum-norm point on a line: for a single linear constraint, the point with smallest L2 norm is where a growing circle first touches the line, tangent to it; for L1 it is where a growing diamond first touches, which favors sparse (axis-aligned) solutions; for L infinity it is where a growing square touches.
• Gram-Schmidt: converts independent columns a1, ..., an of a matrix A into orthonormal columns q1, ..., qn by repeatedly subtracting off the component already covered by earlier q's, then normalizing.
• QR factorization: A = QR, where Q has orthonormal columns and R is upper triangular; because Q is orthogonal, R = Q^T A, so each entry of R is simply the dot product of a Q column with an A column.
• Modified Gram-Schmidt: subtracting off one component at a time (rather than all at once at the end) is mathematically the same as the classical version but more numerically reliable.
• Column pivoting: at each step, compare the remaining piece of every unfinished column (after removing the parts already accounted for) and pick the largest one next, similar to choosing a large pivot in elimination, to avoid dividing by a tiny vector.
• Krylov space: the space spanned by b, Ab, A^2 b, ..., A^(J-1) b for a large sparse matrix A; it is cheap to build because each vector only needs a matrix-vector multiply, but a basis built this way is usually badly conditioned.
• Arnoldi and Lanczos: apply Gram-Schmidt to the Krylov vectors to produce an orthonormal basis; Arnoldi works for a general matrix, Lanczos is the specialized, cheaper version for symmetric matrices.
• Why orthonormal bases matter: if x = Qc and Q has orthonormal columns, each coefficient is just ci = qi^T x, avoiding a full system solve that would otherwise require Q^-1.

Walkthrough

Recap: minimum-norm solutions under different norms (0:01)

Strang begins with administrative notes about the pace of writing his textbook, then returns to a picture from a previous class: minimizing a norm subject to one linear constraint, drawn in two dimensions. For the L2 norm, the unit ball is a circle, so the winning point is where an expanding circle first touches the constraint line, and geometry shows that point is where the circle's radius is perpendicular to the line.

Comparing L1, L2, and L infinity solutions (4:55)

For L1, the unit ball is a diamond, and the first point where the diamond touches the line lands on a corner of the diamond, meaning one coordinate is exactly zero. Strang computes this explicitly for the line 3x1 + 4x2 = 1, then does the same for L2 (using perpendicularity) and L infinity, whose unit ball is a square that touches at a 45-degree corner. As the exponent p increases from 1 to infinity, the minimizing point slides continuously along the line between these three positions, which Strang flags as a natural project topic, including how it generalizes to three dimensions with a plane and a diamond, sphere, or cube.

Gram-Schmidt and the QR factorization (9:12)

Turning to the day's main topic, Strang defines Gram-Schmidt: starting from a matrix A with independent but non-orthogonal columns, the process produces an orthogonal matrix Q whose columns are orthonormal. He shows that the relationship between the two is A = QR, matching the command qr(A) in MATLAB. Rather than deriving R at the end as is traditional, he derives it first: since Q is orthogonal, Q^-1 = Q^T, so R = Q^T A, and each entry of R is just a dot product qi^T aj, which demystifies what R represents.

Working through the Gram-Schmidt steps (15:32)

Strang walks through the construction column by column. q1 is simply a1 normalized to unit length. For q2, he subtracts from a2 its component along q1 ((a2^T q1) q1), leaving a vector orthogonal to q1, then normalizes it. For q3, the same subtraction is repeated against both q1 and q2 before normalizing. He verifies by direct computation that the resulting vector is indeed orthogonal to the earlier q's, confirming the formula is correct.

Column pivoting for numerical stability (23:43)

Just as elimination can fail if a pivot is too small, standard Gram-Schmidt can be numerically dangerous if a column is nearly parallel to an earlier one: subtracting off almost all of it leaves a tiny remaining vector, and dividing by that tiny length amplifies roundoff error. The fix, a more professional Gram-Schmidt, is to compute the remaining, already-orthogonalized piece of every unfinished column at each stage and choose the largest one to become the next q, moving it into place with a permutation. Strang emphasizes this costs no extra work overall, since those partial subtractions have to be computed eventually regardless of the order chosen.

Krylov spaces, Arnoldi, and Lanczos (35:57)

For very large, sparse matrices where inverting A is out of the question, Strang introduces the Krylov space spanned by b, Ab, A^2 b, ..., A^(J-1) b, built cheaply using only repeated matrix-vector multiplication. The goal becomes finding the closest solution to Ax = b within this reduced-dimension space rather than solving exactly. Because the raw Krylov vectors tend to point in nearly the same direction and make a poor basis, they must be orthogonalized, which is exactly Gram-Schmidt applied to this basis. This is the Arnoldi process for a general matrix, or the cheaper Lanczos process when A is symmetric. Strang closes by pointing to Golub and Van Loan's Matrix Computations as the standard reference text covering this material.

Before you watch

• Be comfortable with the basic idea of least squares and orthogonal projections from earlier 18.065 lectures.
• Know how unit balls for L1, L2, and L infinity norms look in two dimensions (diamond, circle, square).
• Review elimination and pivoting, since column pivoting in Gram-Schmidt is presented as a direct parallel.

Check your understanding

1. Why does the minimum-L1-norm solution on a line tend to have a zero coordinate, while the minimum-L2-norm solution generally does not?
2. Starting from A = QR, show why each entry of R equals a dot product between a column of Q and a column of A.
3. Walk through Gram-Schmidt by hand on three simple 2D or 3D vectors, computing each q in order.
4. Why can standard Gram-Schmidt become numerically unstable, and how does column pivoting address that risk?
5. Why is an orthonormal basis for a Krylov space useful for approximating the solution to Ax = b, and why can't the raw Krylov vectors be used directly?