Lecture 32: ImageNet's CNN and the Convolution Rule

What this lecture covers

This lecture opens with the historical AlexNet paper (Krizhevsky, Sutskever, Hinton) that won the 2012 ImageNet competition using a convolutional neural network, as motivation for why convolution matters at scale. It then carefully defines convolution for vectors and functions, cyclic versus non-cyclic convolution, and derives the convolution rule connecting matrix multiplication of circulants to pointwise multiplication of their eigenvalues (the Fourier transform), plus a look ahead to two-dimensional convolution via the Kronecker product.

After watching, you can explain why convolutional layers use far fewer weights than fully connected layers, compute the length of a convolution of two vectors in the cyclic and non-cyclic cases, and state the convolution rule that lets convolution be computed quickly through the fast Fourier transform.

Key ideas

• ImageNet and AlexNet: a large convolutional neural network (60 million parameters, five convolutional and three fully connected layers) trained on 1.2 million images achieved a 15% top-five test error versus 26% for the next-best method in 2012, a landmark result for convolutional nets.
• Why convolution reduces weights: a convolutional layer reuses the same small set of weights along diagonals instead of learning a full weight matrix, which is essential when the number of pixels (and hence potential weights) is enormous.
• Dropout: a regularization technique, used in AlexNet, that randomly removes some connections during training to reduce overfitting in fully connected layers.
• Convolution of vectors and polynomials: convolving two vectors corresponds to multiplying the polynomials whose coefficients are those vectors; the k-th convolution coefficient sums all products c_i * d_j where i + j = k.
• Cyclic vs. non-cyclic convolution: non-cyclic (ordinary) convolution of vectors of length P and Q produces a vector of length P + Q - 1; cyclic convolution wraps indices modulo N, keeping the result at length N, and corresponds to multiplying circulant matrices.
• Eigenvalues from the Fourier matrix: for a circulant matrix C, multiplying the Fourier matrix F by C's defining vector gives C's eigenvalues directly.
• The convolution rule: convolving two vectors and then transforming the result by the Fourier matrix gives the same answer as transforming each vector separately and multiplying the results component-by-component (a Hadamard product); this is why the eigenvalues of a product CD equal the componentwise product of the eigenvalues of C and D.
• Why the rule matters: because the Fourier transform can be computed in N log N time by the FFT, using the convolution rule (transform, multiply, and optionally transform back) is much faster than direct convolution, which costs N^2.
• Kronecker product: the standard way to build a two-dimensional operator (such as a 2D Laplacian) from two one-dimensional matrices, producing an N^2-by-N^2 matrix for an image with N^2 pixels.

Walkthrough

The AlexNet result on ImageNet (0:00)

The lecture presents the AlexNet paper's abstract: a large convolutional neural network trained for five days on two GPUs, classifying 1.2 million high-resolution images, achieved a top-five error rate of 15% versus 26% for the runner-up in the 2012 competition. It notes the network's scale (60 million parameters, 650,000 neurons, five convolutional layers, three fully connected layers), the use of dropout to reduce overfitting in the fully connected layers, and that convolutional layers need far fewer weights than fully connected ones.

Convolution of vectors and functions (5:56)

Building from the polynomial multiplication view, the lecture derives the formula for the k-th component of a convolution C * D as a sum over all index pairs (i, j) with i + j = k. It extends this to a convolution of two continuous functions, defined as an integral, (f * g)(x) = integral f(t) g(x - t) dt, drawing the parallel between the discrete and continuous cases.

Cyclic convolution and Toeplitz vs. circulant matrices (9:13)

The lecture distinguishes non-cyclic convolution (associated with Toeplitz matrices, with constant but non-wrapping diagonals) from cyclic convolution (associated with circulant matrices, with wrapping diagonals). It works out that a non-cyclic convolution of vectors of length P and Q has P + Q - 1 components, while a cyclic convolution of two length-N vectors stays at length N by folding indices modulo N.

Eigenvalues of circulants via the Fourier matrix (21:22)

Recalling that the eigenvectors of any circulant are columns of the Fourier matrix F, the lecture shows that multiplying F by a circulant's defining vector produces that circulant's eigenvalues directly, since a circulant is a combination of powers of the shift matrix P and its eigenvalues combine the same way.

The convolution rule (25:19)

The lecture states and justifies the convolution rule: convolving two vectors and then transforming by F gives the same result as transforming each vector separately and multiplying the transforms component by component. The proof rests on the fact that commuting circulant matrices share eigenvectors, so the eigenvalues of a product are the componentwise product of the individual eigenvalues.

Why the rule matters: speed via the FFT (31:59)

Comparing costs, the lecture shows that direct convolution of length-N vectors takes N^2 operations, while using the FFT to transform, multiply component-wise, and (if needed) transform back takes on the order of N log N, making the convolution rule far faster for large N, with applications such as multiplying large integers.

Two-dimensional convolution and the Kronecker product (32:20)

Extending to two dimensions, the lecture defines the 2D convolution integral and introduces the Kronecker product (MATLAB's kron) for building an N^2-by-N^2 two-dimensional operator from two N-by-N one-dimensional matrices, illustrated with a 2D Laplacian built as a Kronecker sum of two second-derivative matrices.

Before you watch

• Watch the previous two lectures on circulant matrices and on the eigenvectors of circulant matrices (the Fourier matrix), since this lecture builds directly on both.
• Review polynomial multiplication, since convolution is introduced as an analogy to multiplying polynomials.
• Be comfortable with matrix eigenvalues and eigenvectors, and with the basic cost of matrix and vector operations.

Check your understanding

1. Why does a convolutional layer need far fewer weights than a fully connected layer when processing images?
2. How does the length of a convolution differ between the cyclic and non-cyclic cases, and why?
3. What does the convolution rule say about the relationship between convolving two vectors and multiplying their Fourier transforms?
4. Why does the existence of the fast Fourier transform make the convolution rule computationally useful?
5. What is the Kronecker product used for when moving from one-dimensional to two-dimensional convolution?