Lecture 1: The Spelled-Out Intro to Neural Networks and Backpropagation

What this lecture covers

This lecture answers a single question: what actually happens when a neural network learns? Instead of using PyTorch as a black box, Andrej Karpathy builds micrograd, a scalar-valued automatic differentiation engine, from an empty Jupyter notebook. By the end you will have implemented backpropagation and a small neural network library yourself, in well under 200 lines of Python, and understand what every .backward() call in a real deep learning framework is doing underneath.

As the opening lecture of the series, it assumes only basic Python and a rough memory of calculus. Everything later in the course, including transformers and GPT, is built on the same mechanics introduced here: wrapping numbers in objects that track how they were produced, and using the chain rule to push a gradient backward through that record. After watching, you should be able to explain what a derivative means for a function of several variables, build a tiny expression graph by hand, and implement forward and backward passes for basic operations.

Key ideas

• Value object: a wrapper around a Python scalar that also stores its gradient, the operation that produced it, and pointers to the values that fed into it, so an entire expression can be reconstructed as a graph.
• Derivative as sensitivity: (f(x+h) - f(x)) / h as h shrinks tells you how much a function's output responds to a tiny nudge of one input; this numerical estimate is used throughout to check the manual math.
• Chain rule: the derivative of a composed function is the product of the derivatives of its parts, which is what lets a gradient be passed backward one local step at a time.
• Local gradients: every operation (add, multiply, tanh, power) only needs to know its own local derivative; the chain rule combines these local pieces into the derivative of the final output with respect to any earlier value.
• Gradient accumulation: when a value is reused in multiple places, its incoming gradients must be summed (+=), not overwritten, otherwise earlier contributions are lost.
• Topological sort: nodes must be processed in an order where every node comes after everything it depends on, so backpropagation only visits a node once all of its dependents have already propagated their gradient to it.
• Neuron, layer, MLP: a neuron is a weighted sum of inputs plus a bias passed through a nonlinearity like tanh; a layer is a set of independent neurons; a multi-layer perceptron chains layers sequentially.
• Gradient descent: after computing the loss and calling backward(), every parameter is nudged a small step in the negative direction of its gradient, which is repeated to gradually reduce the loss.

Walkthrough

Micrograd overview and the Value object (0:25)

The lecture opens by demonstrating finished micrograd: wrapping numbers in Value objects, combining them with operators like +, *, and **, and then calling .backward() to fill in a .grad attribute on every value in the expression. This previews the whole lecture before anything is built.

The meaning of a derivative (8:08)

Using a plotted parabola, the lecture works through the limit definition of a derivative numerically rather than symbolically, since no one writes out the symbolic derivative of an entire neural network by hand. It extends this to a function of three inputs, checking by hand how nudging each input changes the output.

Building the Value object and expression graphs (19:09)

The Value class is built up piece by piece: wrapping a number, then adding __add__ and __mul__ so values can be combined, then tracking _prev (child values) and _op (the operation that created a value). A graph-drawing helper visualizes the resulting expression trees.

Manual backpropagation on a small expression (32:10)

Working backward from an output L, the lecture fills in gradients by hand using the chain rule, showing that a + node simply routes its incoming gradient to both children unchanged, while a * node multiplies the incoming gradient by the other operand's value. Each manual result is checked against the numerical derivative.

Backpropagating through a neuron (52:52)

A single neuron is modeled as tanh(w1*x1 + w2*x2 + b). Because tanh cannot be built from only addition and multiplication, its local derivative 1 - tanh(x)^2 is derived from its definition and implemented directly, then gradients are propagated back to the weights by hand.

Automating backward with _backward and topological sort (1:09:02)

Each operation is given a _backward closure that performs its local piece of the chain rule and accumulates (rather than overwrites) gradients into its inputs. A topological sort orders all nodes in an expression so that backward() can call every node's _backward in the correct reverse order automatically. The lecture also fixes a real bug: when a value is used more than once, gradients must add up, not overwrite.

Building neuron, layer, and MLP classes (1:43:55)

The Value engine is used to build a Neuron, a Layer of neurons, and a multi-layer perceptron (MLP) that chains layers together, mirroring the structure and parameters() method of PyTorch's nn.Module.

Training loop and gradient descent (2:01:12)

A tiny dataset of four examples is used with a mean-squared-error loss. The lecture runs forward pass, backward(), and a parameter update in a loop, including a deliberate demonstration of the common bug of forgetting to zero out gradients before each backward call, and shows the corresponding operations already implemented inside the real micrograd and PyTorch source code.

Before you watch

• Be comfortable with basic Python, including classes, operators, and list comprehensions.
• Recall the high-school definition of a derivative and the chain rule; the lecture re-derives both from scratch, so no more than a vague memory is required.
• This is the first lecture in the series, so no prior lecture in this course is assumed.

Check your understanding

1. Why does an addition node simply pass its output gradient unchanged to both of its children during backpropagation?
2. Why must gradients be accumulated with += rather than overwritten when a value is used more than once in an expression?
3. What role does topological sort play in making sure backward() produces correct gradients?
4. How does the local derivative of tanh(x) get combined with an incoming gradient during the backward pass of a neuron?
5. What happens to training if you forget to zero out each parameter's gradient before calling backward() again?