Lecture 36: Alan Edelman and Julia Language

What this lecture covers

This is the final class session, a guest lecture by Alan Edelman, co-creator of the Julia programming language, given after Gilbert Strang leaves the room. Edelman first closes out a proof Strang left unfinished about the zero matrix and rank, then spends most of the hour making the case that the choice of programming language matters mathematically, not just for convenience, and that automatic differentiation is neither symbolic nor numerical differentiation but a third, distinct technique.

The core demonstration walks through forward-mode automatic differentiation by hand, using an eight-line Julia implementation of the ancient Babylonian square-root algorithm. Edelman then sketches, on the blackboard, how backward-mode automatic differentiation (backpropagation in a neural network) reduces to solving a triangular linear system, tying the whole course's linear algebra back to how neural networks are actually trained. After watching, you should understand what a "dual number" is, why overloading arithmetic operators lets a language differentiate code without symbolic or numerical methods, and why backpropagation is just linear algebra in disguise.

Key ideas

• Zero-matrix rank proof: the independent columns of a zero matrix form an empty 3-by-0 matrix, showing that the row-rank-equals-column-rank proof still holds in that degenerate case.
• Dual number: a pair of floats representing a function value and its derivative together; arithmetic operators are overloaded so that computations on dual numbers automatically carry derivative information.
• Forward-mode automatic differentiation: computing a derivative by propagating a dual number through the same code used to compute the function, using only the sum, product, and quotient rules at each step, with no symbolic expressions and no finite differences.
• Why it's a third method: automatic differentiation is neither symbolic differentiation (which explodes in size, as shown with a Python symbolic package) nor numerical finite differencing (which trades off truncation and roundoff error); it computes exact derivatives step by step through the algorithm itself.
• Structured types over sparse storage: a SymTridiagonal type in Julia stores only the diagonal and off-diagonal vectors, avoiding both dense storage and generic sparse-matrix overhead, letting operations match the underlying mathematics.
• Backpropagation as linear algebra: writing the chain rule through a scalar neural network layer by layer produces a system (I - L) dX = dP, so back-propagating derivatives is exactly a lower-triangular linear solve, dX = (I - L)^{-1} dP.
• Machine learning as optimization: training a network is fundamentally a large minimization problem, and automatic differentiation is the mechanism that makes computing the needed gradients practical.

Walkthrough

Finishing the zero-matrix rank proof (0:01)

Edelman opens by picking up a proof Strang had demonstrated earlier: that row rank equals column rank, shown by placing the independent columns of a matrix into a matrix of their own. He works through what happens when the original matrix is the zero matrix, concluding that its independent columns form an empty 3-by-0 matrix, and that multiplying it out still produces a 3-by-3 zero matrix, so the proof holds even in this edge case.

Why Julia, and Google's language filtering (3:02)

Edelman describes a Google blog post that filtered candidate languages for machine learning on technical merit and usability, eliminating Python, Java, C++, and Rust, and leaving only Julia and Swift as viable. He uses this to motivate why the language itself, not just the algorithm, matters for machine learning work, and briefly discusses the psychology of programming-language adoption.

Forward-mode automatic differentiation with dual numbers (6:06)

The lecture's central demonstration: Edelman builds the Babylonian square-root algorithm (t = (t + x/t) / 2, iterated) in Julia, then defines an eight-line Dual type holding a function value and derivative pair, overloading addition and division with the sum and quotient rules. Running the unmodified Babylonian code on a dual number produces both the square root and its exact derivative, without ever writing 1/2 * x^(-1/2) or using finite differences.

Comparing to symbolic computation and inspecting assembler (19:14)

To show what automatic differentiation is not, Edelman runs the same algorithm through a Python symbolic package, producing iterates with huge, memory-hungry coefficients. He contrasts this with the compiled assembler for the dual-number version, which is short and efficient, arguing that Julia's approach gets both correctness and performance that a purely symbolic or naively translated approach would not.

Deriving forward-mode differentiation line by line (22:20)

Edelman explicitly differentiates each line of the Babylonian algorithm using the quotient rule, showing that running this derivative code alongside the original produces the same result the dual-number type gave automatically. He explains that a JIT compiler doing this operator overloading automatically is the modern replacement for older Fortran source-to-source translators that generated derivative code.

Structured types: tridiagonal matrices (28:24)

Returning to a matrix example, Edelman builds a symmetric tridiagonal "Strang matrix" using Julia's SymTridiagonal type, which stores only the diagonal and off-diagonal vectors rather than full dense or generic sparse storage. He argues this lets operations like a linear solve use a specialized, efficient algorithm that matches the matrix's actual structure.

Backpropagation as a triangular solve (30:26)

In the closing minutes, Edelman sets up a scalar neural network with weights, biases, and an activation function (mentioning the sigmoid and ReLU), then differentiates the forward computation layer by layer. He assembles the result into block-matrix form, showing that the derivatives satisfy (I - L) dX = dP for a lower-triangular matrix L, so backpropagation is simply solving that triangular system, a computation linear algebra libraries already implement efficiently.

Before you watch

• Be comfortable with the chain rule, product rule, and quotient rule from calculus, since the lecture builds automatic differentiation directly from them.
• Review the course's earlier treatment of the SVD and matrix factorizations as changes of variables, which Edelman references as his own motivation for studying automatic differentiation.
• Some familiarity with basic neural network structure (weights, biases, activation functions) helps for the closing backpropagation section.
• Knowing what a tridiagonal or structured matrix is (covered earlier in the course) makes the Julia type example clearer.

Check your understanding

1. Why does representing a number as a (value, derivative) pair let ordinary arithmetic code compute derivatives automatically?
2. How does forward-mode automatic differentiation differ from both symbolic differentiation and numerical finite differences?
3. What does storing only the diagonal and off-diagonal vectors of a SymTridiagonal matrix save compared to dense or generic sparse storage?
4. In the backpropagation derivation, what does the matrix L represent, and why does solving (I - L) dX = dP correspond to back-propagating gradients?