PyTorch Tutorial (CS224N Review Session)

What this lecture covers

This is a teaching-assistant review session that introduces PyTorch, the deep learning framework used for the course's later assignments, to students who already know NumPy. It builds up from raw tensor operations to a full training loop, so that afterward you can define and train a small neural network in PyTorch without needing to write your own backpropagation or optimizer code.

The session covers four building blocks in order: tensors (creation, shapes, reshaping, and indexing, deliberately parallel to NumPy), autograd (PyTorch's automatic differentiation system), neural network modules (torch.nn layers such as linear layers and activations), and optimization (the torch.optim package and the standard training loop). It closes with a worked example that strings all four pieces together to fit a small model.

Key ideas

• Tensors as multi-dimensional arrays: a PyTorch tensor is the framework's equivalent of a NumPy array, created with torch.tensor() from a list of lists, with a configurable data type (dtype) and support for element-wise operations and NumPy-style broadcasting.
• Shape-first debugging: printing a tensor's .shape at each step is described as the most reliable debugging tool, more trustworthy than PyTorch's error messages, since operations can silently reshape data internally.
• Reduction operations collapse a dimension: calling sum(), mean(), or similar functions with a dim argument removes that dimension from the result; with no dim given, the operation runs over the whole tensor.
• Indexing mirrors NumPy: slicing with :, indexing with a list of positions, and combining indices across multiple dimensions all work the same way as in NumPy arrays.
• Autograd: setting requires_grad=True on a tensor makes PyTorch track operations on it; calling .backward() computes gradients via the chain rule and stores them in .grad.
• Gradient accumulation: PyTorch adds new gradients to .grad rather than overwriting them, which is why a training loop must call optimizer.zero_grad() before each backward pass.
• nn.Module building blocks: layers like nn.Linear, activation functions, and nn.Sequential let you compose a network from existing components instead of writing matrix operations by hand; a custom network subclasses nn.Module and defines __init__ and forward.
• The training loop pattern: zero the gradient, run a forward pass, compute the loss, call loss.backward(), then call optimizer.step(), repeated once per epoch.

Walkthrough

Tensors as the building block of PyTorch (0:05)

The instructor frames the session around four topics: tensors, autograd, neural network modules, and optimization. Tensors are introduced as PyTorch's version of NumPy arrays, used to represent everything from a single image (as a width-by-height tensor) to a batch of images with channels (a four-dimensional tensor).

Creating and manipulating tensors (5:10)

The lecture covers torch.tensor() for creating tensors from Python lists, torch.zeros, torch.ones, and torch.arange for quickly instantiating tensors of a given shape or range, and notes that arithmetic operations are element-wise by default. It explains that PyTorch's broadcasting rules match NumPy's: dimensions are compatible if they are equal or if one of them is 1, and notes that variable-length data, such as sentences of different lengths, is typically handled by padding to a fixed shape rather than using ragged tensors.

Shapes, reshaping, and debugging (11:15)

The instructor emphasizes printing .shape as the primary debugging habit, then demonstrates reshape() for changing a tensor's dimensions (for example, turning a flat 15-element tensor into a 5-by-3 tensor) and distinguishes view(), which returns a view of the same underlying data, from reshape(). It also shows converting between NumPy arrays and PyTorch tensors, and demonstrates reduction operations like sum() and mean(), explaining that the dim argument specifies which dimension gets collapsed in the result.

Indexing tensors (20:21)

Working through a three-dimensional example tensor, the lecture shows how indexing a single position along one dimension collapses that dimension, how a : copies an entire dimension, how to slice ranges of rows, and how list indexing (indexing with a list of positions, such as [0, 2, 4]) selects multiple elements at once. It closes with .item() for extracting a Python scalar from a one-element tensor, which is needed because a loss value must be a scalar before computing gradients.

Autograd and gradient accumulation (26:27)

Autograd is presented as PyTorch's automatic differentiation system: tensors created with requires_grad=True have their operations tracked, and calling .backward() on a downstream value computes gradients via the chain rule, storing them in the tensor's .grad attribute. A live example with y = 3x^2 shows the gradient evaluating to 6x, then a second .backward() call on a related quantity shows the gradient value doubling, because PyTorch accumulates gradients rather than overwriting them, which is why a training loop must explicitly zero the gradient before each new backward pass.

Building neural network modules (34:38)

The lecture introduces torch.nn, starting with nn.Linear(input_dim, output_dim), which applies an affine transformation to the last dimension of its input and includes a bias term by default. It covers reading and matching tensor shapes when stacking layers, other common layers (convolutions, batch normalization, pooling), composing layers with nn.Sequential, and defining a custom network by subclassing nn.Module with an __init__ method (which defines the layers) and a forward method (which defines the computation).

Training loop and optimization (43:48)

The final section introduces torch.optim and walks through the standard PyTorch training loop: for each epoch, call optimizer.zero_grad(), run a forward pass to get predictions, compute a loss (using a built-in loss function such as cross-entropy), call loss.backward() to compute gradients, and call optimizer.step() to update parameters. Running this loop on a small example shows the training loss dropping over epochs as the model fits the data.

Before you watch

• Familiarity with NumPy arrays, shapes, and broadcasting (covered in this course's Python tutorial) is assumed throughout.
• A basic understanding of what a neural network's forward pass, loss, and gradient are will make the autograd and training-loop sections easier to follow.

Check your understanding

1. Why does printing a tensor's shape at each step help more with debugging than relying on PyTorch's error messages?
2. What is the difference between calling .sum(dim=0) and .sum(dim=1) on a two-dimensional tensor, and what shape does each produce?
3. Why does PyTorch accumulate gradients in .grad instead of overwriting them, and what does this require you to do in a training loop?
4. What are the two methods you must define when creating a custom network by subclassing nn.Module, and what does each one do?
5. Walk through the five steps of a standard PyTorch training loop, in order, and explain what each step accomplishes.