Lecture 14: Implementing Convolutions

What this lecture covers

This lecture answers a concrete question left over from the math of convolutions: how do you actually code one? It follows the Deep Learning Systems course's earlier lecture on the mathematics of convolution and automatic differentiation, moving from theory into a live-coded notebook. The instructor builds the same convolution operator four times, each version faster than the last, ending with a single matrix multiplication.

After watching, you should be able to explain how image and kernel tensors are laid out in memory, implement a correctness-checked convolution with nested loops, replace those loops with batched matrix multiplies, and understand how NumPy's as_strided call builds an im2col-style view without explicit copying.

Key ideas

• NHWC layout: the lecture stores images as batch, height, width, channel, arguing this ordering makes the matrix-multiply view of convolution more natural than PyTorch's NCHW default.
• Reference implementation: before writing a new operator, borrow a trusted implementation (here, PyTorch's conv2d) to check your own output against, even in a course whose goal is to avoid using PyTorch.
• Naive convolution: a seven-nested-loop implementation over batch, channels, output position and kernel position; correct but around 10,000 times slower than PyTorch on a small test case.
• Convolution as matrix multiply: a 1x1 convolution is exactly a matrix multiply between the flattened image and the kernel; a general kxk convolution can be built by summing kxk shifted matrix multiplies, cutting the implementation to two loops.
• Strides: NumPy arrays are stored as flat contiguous memory plus a stride per dimension; understanding strides is what makes the later tricks possible.
• as_strided: a NumPy function that reinterprets existing memory with new shape and strides, letting you view a matrix as overlapping tiles without copying any data.
• ascontiguousarray: copies a strided view into new contiguous memory; needed because as_strided views alias overlapping data and can't be reshaped for a matmul until they're materialized.
• im2col: extracting every overlapping kxk patch of the image as a matrix (via as_strided plus reshape) turns the whole convolution into one matrix multiplication, at the cost of duplicating memory for overlapping patches.

Walkthrough

Storage order for images and weights (1:01)

The lecture opens by choosing how convolution inputs are stored. Images become rank-4 tensors ordered batch, height, width, channel (NHWC), and kernel weights become rank-4 tensors ordered kernel height, kernel width, input channels, output channels. The instructor argues this is preferable to PyTorch's NCHW/out-in-kernel-kernel layout because it makes the channel dimension line up naturally with matrix multiplication.

A reference implementation and the naive for-loop version (9:09)

A helper function wraps PyTorch's conv2d, permuting tensors between the course's NHWC format and PyTorch's format, to serve as a correctness check. The instructor then writes the simplest possible convolution: nested loops over batch, input channel, output channel, output row, output column, and the two kernel dimensions. It matches the reference numerically but takes about seven seconds for one small batch, versus under a millisecond for PyTorch.

Convolution as matrix multiplication (24:24)

Starting from the observation that a 1x1 convolution is just a matrix multiply, the lecture generalizes: a kxk convolution is the sum, over each kernel position, of a matrix multiply between a shifted slice of the image and the corresponding kernel slice. This collapses the loop nest to two loops over kernel positions and roughly triples speed, but still leaves Python-level looping.

Strides and the as_strided call (37:36)

The lecture explains how arrays are stored contiguously with per-dimension strides, then introduces numpy.lib.stride_tricks.as_strided, which reinterprets an array's existing memory with a new shape and new strides. A worked example tiles a 6x6 matrix into 3x3x2x2 blocks purely by choosing strides, with no copying, and shows that the result aliases the original memory until ascontiguousarray is called to actually copy it into the new layout.

Building im2col for multi-channel convolution (57:53)

The same as_strided trick is applied to extract every overlapping kxk window of an image as a 6th-order tensor, which is then reshaped into a matrix and multiplied by the flattened kernel weights in one call. The instructor stresses that this reshape (via ascontiguousarray) genuinely expands memory, since each pixel appears in multiple overlapping patches, unlike the earlier tiling example where memory size stayed fixed.

Timing comparison and final notes (1:09:04)

The im2col version runs in about 14 milliseconds versus 27 for the two-loop matmul version and about 4 for PyTorch, roughly doubling speed over the previous approach while remaining a handful of lines of NumPy. The lecture closes by noting that implementing as_strided and ascontiguousarray (called "compact") from scratch, as required in the course's homework, is nontrivial even though using them here looks effortless.

Before you watch

• Review the prior lecture's treatment of convolution mathematics and its expression via automatic differentiation.
• Be comfortable with NumPy reshaping, matrix multiplication broadcasting over leading dimensions, and basic stride/row-major storage concepts.

Check your understanding

1. Why does the lecture argue that NHWC is a better storage order for this implementation than PyTorch's NCHW?
2. How can a kxk convolution be expressed as a sum of matrix multiplies over kernel positions?
3. What is the difference between what as_strided does and what ascontiguousarray does?
4. Why does the im2col approach use more memory than the original image, even though as_strided itself doesn't copy anything?