Lecture 10: Convolutional Networks

What this lecture covers

The lecture asks why images are not simply flattened into vectors and fed to fully connected layers, and answers that this wastes parameters and ignores structure images actually have, such as shift invariance. It builds up the convolution operator from scratch: local weight sharing across spatial positions, multiple input and output channels, and the practical additions (padding, pooling, strided convolutions, grouped convolutions, dilations) used in real architectures.

The lecture then turns to automatic differentiation, since convolutions must work as atomic, differentiable operators in a deep learning library. It shows that the transpose of a convolution is itself a convolution with a flipped filter, and that convolutions can also be computed via an "im2col" matrix expansion, which explains why convolution is implemented as an operator rather than built from matrix multiplications directly. After watching, you should be able to explain how a convolution reduces parameter count compared to a fully connected layer, describe the common convolution variants, and explain conceptually how gradients flow backward through a convolution.

Key ideas

• Local weight sharing: a convolution restricts each hidden unit to a local receptive field and reuses the same small filter across all spatial locations, drastically cutting parameters compared to a fully connected layer over an image.
• Convolution vs. correlation: what deep learning calls a convolution is what signal processing calls a correlation; the true (flipped) convolution corresponds to multiplying by the transpose of the correlation operator.
• Multi-channel convolutions as matrix-vector products: each spatial position in a multi-channel convolution can be viewed as a vector in the channel dimension, and each filter position as a matrix mapping input channels to output channels.
• Padding: zero-padding an input by (k-1)/2 on each side (for odd kernel size k) keeps the output the same spatial size as the input, which is the near-universal default in practice.
• Downsampling: max/average pooling and strided convolutions both reduce spatial resolution as a network goes deeper, for computational and representational reasons.
• Grouped and depthwise convolutions: restricting which input channels connect to which output channels reduces parameter count sharply when channel counts are very large.
• Dilations: spreading a filter's sampled positions apart increases the receptive field without increasing the number of weights, at the cost of needing more padding.
• Transpose of a convolution: multiplying by the transpose of a convolution operator is equivalent to convolving with the filter flipped, which is how gradients with respect to the input are computed without ever forming the equivalent large sparse matrix.

Walkthrough

Why flattening images does not scale (2:11)

The lecture contrasts MNIST, small enough to flatten into a 784-element vector, with a 256x256 RGB image, which would need roughly 200,000 input dimensions and hundreds of millions of parameters for even one fully connected hidden layer. Beyond the parameter count, flattening discards the fact that a shifted image is still recognizably the same image, motivating a network structure that captures spatial locality and shift invariance directly.

The convolution operator (5:14)

Convolutions are built from two premises: hidden units are arranged spatially like the input, and the same small filter (for example 3x3) is applied at every location by sliding it across the image, computing each output as an inner product of the filter with the corresponding local patch. This gives large parameter savings (nine weights instead of millions for a single channel) and directly captures shift invariance, since the same weights apply everywhere in the image.

Convolutions in image processing and multi-channel convolutions (13:25)

Before their use in deep learning, convolutions with fixed, hand-designed filters were used for tasks like Gaussian blurring and edge detection via image gradients; deep networks instead learn the filter values. Real networks operate on multi-channel tensors (height, width, channels), and the lecture shows that a multi-channel convolution is best understood by treating each spatial position as a vector over channels and each filter entry as a matrix mapping input channels to output channels, rather than writing out the full summation explicitly.

Practical elements: padding, pooling and strides (26:02)

Because a convolution without padding shrinks the output relative to the input, zero-padding by (k-1)/2 on each side (for odd kernel sizes) is used almost universally to preserve spatial size. To progressively reduce resolution deeper in the network, either pooling (max or average over small blocks) or strided convolutions (sliding the filter by more than one position at a time) are used, both halving resolution in each spatial dimension for a stride or pool size of 2.

Grouped convolutions and dilations (33:15)

When channel counts get very large, group convolutions restrict which input channels feed which output channels, cutting parameter count; the extreme case, where each group has one channel, is called a depthwise convolution. Dilated convolutions spread out the positions a filter samples so a single layer covers a larger receptive field without adding parameters, though this requires additional padding to keep output size matched to the input.

Differentiating convolutions: the transpose is a flipped convolution (38:28)

To integrate convolutions into an automatic differentiation library as an atomic operator, the lecture derives the backward pass by first recalling that for a matrix-vector product z = Wx, multiplying an adjoint by the transpose of W gives the gradient with respect to x. Writing a 1D convolution explicitly as multiplication by a banded matrix of the filter weights, the lecture shows that transposing this matrix produces the same banded structure with the filter reversed — so the gradient with respect to the input is computed simply by convolving the incoming adjoint with the flipped filter, without ever constructing the large sparse matrix.

Computing the gradient with respect to the weights: im2col (59:57)

The same convolution can also be written as a matrix built from shifted copies of the input, multiplied by the filter as a vector; constructing this matrix is called im2col. This form makes the gradient with respect to the weights easy to express as a matrix product against that same expanded input matrix. Despite duplicating memory, explicitly building the im2col matrix and running one large matrix-matrix multiply turns out to be an efficient way to implement convolutions in practice, as long as it happens inside the operator rather than in the computational graph.

Before you watch

• Be comfortable with how automatic differentiation computes adjoints and backward passes through operators, covered in earlier lectures on building the needle library.
• Review matrix-vector product differentiation (dz/dx = W, and multiplying adjoints by W^T), which this lecture uses as the template for deriving convolution gradients.
• Familiarity with how fully connected layers and their parameter counts work will help motivate why convolutions are needed for images.

Check your understanding

1. Why does flattening a 256x256 RGB image into a vector for a fully connected layer become impractical, and how does a convolution avoid this?
2. What is the difference between what deep learning calls a "convolution" and what signal processing calls a "convolution"?
3. How does zero-padding by (k-1)/2 on each side keep a convolution's output the same size as its input, and why must k be odd for this formula?
4. Explain, in your own words, why multiplying by the transpose of a convolution operator is equivalent to convolving with a flipped filter.
5. Why is the im2col matrix constructed as part of the convolution operator's implementation rather than as separate operations in the computational graph?