Lecture 5: Convolutional Neural Networks

What this lecture covers

This lecture answers a specific question: how do you build a network that respects the spatial structure of an image instead of flattening it into a long vector? Lecture 4 built up fully connected layers and backpropagation; this lecture replaces the fully connected layer with the convolutional layer, the building block that gives convolutional neural networks (ConvNets) their name.

The lecture opens with a history of neural networks from the 1957 perceptron through backpropagation in 1986 to the 2012 AlexNet result that made ConvNets mainstream, then connects this history to Hubel and Wiesel's experiments on the cat visual cortex. It then works through the mechanics of convolution, stride, padding, and pooling in detail, including several worked examples of output-size arithmetic. After this lecture you should be able to compute the output volume and parameter count of a convolutional layer, explain why weight sharing and local connectivity make ConvNets efficient, and describe how stacking Conv, ReLU, and pooling layers builds up a hierarchy of features.

Key ideas

• Convolutional layer: a small filter (e.g. 5x5x3) slides across the width and height of the input, computing a dot product at each location, but always extends through the full input depth.
• Weight sharing: the same filter weights are reused at every spatial location, unlike a fully connected layer where every output neuron has its own full set of weights.
• Activation map: the 2D grid of outputs produced by sliding one filter over the input; using K filters produces a volume with depth K.
• Output size formula: for input size N, filter size F, and stride S, the output spatial size is (N - F) / S + 1.
• Zero padding: padding the input border with zeros lets filters be centered on edge pixels and keeps the output size from shrinking every layer.
• Receptive field: the region of the input that a given filter covers; a 5x5 filter gives a neuron a 5x5 receptive field.
• Pooling layer: downsamples each activation map spatially (commonly with max pooling) without changing the depth, reducing parameters and computation.
• Feature hierarchy: early convolutional layers tend to respond to edges, middle layers to corners and blobs, and later layers to more complex, object-like patterns.

Walkthrough

From perceptrons to AlexNet (1:09)

The lecture traces the lineage of neural networks: Rosenblatt's 1957 perceptron used a simple weight-update rule with no real backpropagation; Widrow and Hoff's 1960 Adaline/Madaline stacked linear layers for the first time; Rumelhart introduced backpropagation with the chain rule in 1986; and Hinton and Salakhutdinov's 2006 work showed deep networks could be trained, though it still needed a layer-by-layer pretraining stage. The turning point came in 2012, when Hinton's lab produced strong speech-recognition results and Alex Krizhevsky's AlexNet dramatically cut the error rate on ImageNet classification, kicking off widespread use of ConvNets.

Hubel and Wiesel and the roots of ConvNets (5:13)

The lecture connects this history to biology. Hubel and Wiesel's experiments on the cat visual cortex found a topographic mapping (nearby cortical cells respond to nearby regions of the visual field) and a hierarchy of cell types: simple cells respond to oriented edges, complex cells add invariance to movement, and hypercomplex cells respond to more specific patterns like corners. Fukushima's 1980 neocognitron modeled this simple/complex cell alternation directly, and Yann LeCun's 1998 network applied backpropagation to a similar architecture for digit and zip-code recognition, a direct ancestor of AlexNet.

Where ConvNets are used today (9:18)

The lecture surveys applications beyond classification: image retrieval, object detection with bounding boxes, pixel-level segmentation, face recognition, video classification, pose estimation, game playing, medical image diagnosis, and generative uses like DeepDream and neural style transfer. The point is that the same convolutional building block underlies a very wide range of vision tasks.

The convolutional layer explained (14:25)

Where a fully connected layer stretches a 32x32x3 image into one vector and multiplies it by a full weight matrix, a convolutional layer keeps the 3D structure and slides a small filter, such as 5x5x3, across it. At each position the filter and the underlying image patch are multiplied element-wise and summed (plus a bias), equivalent to a dot product between the flattened filter and flattened patch. Filters always span the full input depth. Using multiple filters in one layer produces multiple activation maps stacked into an output volume.

Stacking filters into a hierarchy (23:43)

A ConvNet stacks Conv layers, each with many filters, interspersed with activation functions like ReLU. Because each layer's input is the previous layer's output rather than the raw image, later layers effectively reason over earlier feature maps. This produces the same simple-to-complex hierarchy Hubel and Wiesel observed biologically, without being explicitly engineered: early layers pick up edges, middle layers combine these into corners and blobs, and later layers respond to more complex, higher-level patterns.

Spatial arithmetic: stride, padding, and output size (30:56)

Through worked examples, the lecture derives the output-size formula (N - F) / S + 1 for input size N, filter size F, and stride S, and shows that some stride choices don't divide evenly and simply aren't used. It then introduces zero padding to keep the output size from shrinking layer after layer and to let filters process the image's edges and corners properly; common practice is to pad so the output matches the input size (for example, pad by 1 for a 3x3 filter, by 2 for 5x5). Worked examples also cover computing the parameter count of a layer, including the often-missed detail that each filter's depth and bias term both count.

Pooling and the full ConvNet architecture (55:33)

Pooling layers downsample each activation map spatially, most commonly with max pooling over small regions like 2x2 with stride 2, without touching the depth. The lecture explains max pooling as detecting how strongly a filter fired anywhere within a region, and notes some newer architectures use strided convolutions instead of pooling to downsample. The lecture closes by assembling the full pattern: repeated Conv-ReLU blocks with occasional pooling, followed by one or a few fully connected layers and a softmax to produce class scores, pointing ahead to architectures like ResNet and GoogLeNet.

Before you watch

• Be comfortable with the fully connected layer and backpropagation from the previous lecture, since the convolutional layer is introduced as a variant of it.
• Review basic matrix/vector dot products; the convolution operation is explained as a sliding dot product.
• Some familiarity with the ImageNet classification task helps, since AlexNet's result is a key reference point.

Check your understanding

1. Why does a convolutional filter extend through the full depth of the input volume but only cover a small spatial region?
2. Given a 9x9 input, a 3x3 filter, stride 1, and no padding, what is the output size, and how does the formula change if you zero pad the input by 1 on each side?
3. Why does weight sharing across spatial locations reduce the number of parameters compared to a fully connected layer?
4. How does max pooling differ from a strided convolution as a way of downsampling an activation map?
5. In what sense does the hierarchy of features learned by a ConvNet echo the simple and complex cells Hubel and Wiesel found in the visual cortex?