Lecture 4: Automatic Differentiation

What this lecture covers

Taught by co-instructor Tianqi Chen, this lecture asks how deep learning frameworks actually compute gradients, having established in prior lectures that manual derivation doesn't scale. It surveys numerical differentiation (useful for gradient checking, not training) and symbolic differentiation (exact but wasteful, since it doesn't reuse shared subcomputations), then introduces the computational graph as the tool used to derive both forward mode and reverse mode automatic differentiation.

The bulk of the lecture works through reverse mode AD by hand on a small example, defining the "adjoint" of each node and showing how to accumulate contributions when a value feeds into multiple downstream nodes. It then contrasts this with the classic backpropagation algorithm covered in Lecture 3, explaining why modern frameworks build an extended computational graph for the gradient computation rather than running backprop in place, including the ability to get gradients of gradients for free. It closes by generalizing the technique from scalars to tensors and briefly to other data structures.

Key ideas

• Numerical differentiation: approximates gradients via finite differences; the centered-difference formula (f(theta + eps*e_i) - f(theta - eps*e_i)) / (2*eps) has lower error than the one-sided version, but is costly (2n function evaluations for n parameters) and used mainly for gradient checking, not training.
• Symbolic differentiation: derives an exact closed-form gradient expression by applying calculus rules, but naively applied it duplicates work across parameters, leading to unnecessary computation that a smarter method avoids.
• Computational graph: a directed acyclic graph where each node is an intermediate computation with defined inputs and outputs; evaluating it in topological order computes the function, and it's the foundation for deriving automatic differentiation.
• Forward mode AD: propagates derivatives with respect to one input forward through the graph; efficient when there are few inputs and many outputs, but requires one pass per input, making it costly for typical deep learning models (many parameters, one scalar loss).
• Reverse mode AD and the adjoint: defines the adjoint of a node as the partial derivative of the final output with respect to that node, computed backward from the output (where the adjoint is 1) toward the inputs; when a node feeds multiple downstream nodes, its adjoint sums the contributions ("partial adjoints") from each path.
• Reverse mode AD by extending the computational graph: rather than computing concrete adjoint values immediately, the algorithm builds new graph nodes representing the adjoint computations, producing a graph that can be evaluated at any input and reused, and that naturally supports gradients of gradients.
• Reverse mode AD vs. backpropagation: classic backpropagation runs backward operations in place on the original graph with no new graph created; reverse mode AD as computational-graph extension produces a separate, evaluable gradient graph, which modern frameworks (following Theano's lead, unlike earlier ones like Caffe) prefer because it enables higher-order gradients and further compiler-style optimization.
• Generalizing to tensors: the same adjoint definitions extend to matrices and tensors elementwise, giving standard rules like the linear-layer gradient X_adjoint = Z_adjoint @ W^T.

Walkthrough

Why automatic differentiation, and numerical differentiation (0:48)

The lecture recaps that gradient computation is the shared bottleneck across all machine learning algorithms, then introduces numerical differentiation from the calculus definition of a partial derivative, showing why the centered-difference formula is more accurate and why numerical differentiation, despite being costly, remains essential for gradient checking.

Symbolic differentiation and its inefficiency (14:06)

Working through a product-of-parameters example, the lecture shows how manually deriving a symbolic gradient formula wastes computation, since naive symbolic differentiation doesn't reuse intermediate results shared across different partial derivatives, motivating a better approach.

Computational graphs and forward mode AD (18:27)

The lecture defines the computational graph, walks through evaluating one by hand in topological order, then derives forward mode automatic differentiation by propagating a derivative forward from the inputs, noting its inefficiency when there are many input parameters and few outputs (the typical deep learning case).

Deriving reverse mode AD via adjoints (29:45)

The lecture defines the adjoint of a node as the derivative of the scalar output with respect to that node, and derives it recursively backward from the output node (adjoint = 1) through the graph. It works through the multi-pathway case explicitly, showing that when a node feeds multiple outputs, its adjoint is the sum of partial adjoints from each downstream path, and introduces notation for these partial adjoints attached to graph edges.

Implementing reverse mode AD by extending the graph (43:40)

Rather than computing numeric adjoint values directly, the lecture shows building new computational graph nodes that represent each adjoint computation, illustrated on a small example. This produces a reusable gradient graph that can be evaluated at any input, and sets up the comparison with traditional backpropagation.

Reverse mode AD vs. backpropagation, and gradients of gradients (51:50)

The lecture contrasts backpropagation (running backward operations in place on the same graph, as in early frameworks like Caffe) with reverse mode AD as graph extension (pioneered by Theano, adopted by TensorFlow and PyTorch). It highlights two advantages of the graph-extension approach: computing gradients of gradients by simply differentiating the gradient graph again, and more opportunities for compiler-style optimization of the resulting computation.

Generalizing to tensors and other data structures (57:40)

The lecture extends the scalar adjoint definition to matrices and tensors elementwise, deriving the standard linear-layer gradient rule, then briefly notes that adjoints can be generalized to structured data types like dictionaries and tuples, gesturing at the broader idea of differentiable programming.

Before you watch

• Watch Lecture 3 (Parts I and II) first, since this lecture directly compares reverse mode AD to the manual backpropagation derived there.
• Be comfortable with partial derivatives and the multivariable chain rule.
• Having pen and paper to trace through the computational graph examples is recommended, as the lecture itself suggests.

Check your understanding

1. Why is the centered-difference numerical gradient formula more accurate than the one-sided version, and why is numerical differentiation still used despite this cost?
2. Why does forward mode AD become inefficient for typical deep learning training, where there's one scalar loss but many parameters?
3. How is a node's adjoint computed when that node feeds into more than one downstream computation?
4. What is the key difference between traditional backpropagation and reverse mode AD implemented by extending the computational graph, and why does that difference matter for computing gradients of gradients?