Lecture 9: CNN Architectures

What this lecture covers

Building on the previous lecture's tour of deep learning frameworks, this lecture asks a focused question: which convolutional architectures actually won ImageNet, and why did each one work better than what came before? It goes through AlexNet, VGGNet, GoogleNet and ResNet in chronological order, computing output sizes, parameter counts and computational cost by hand for several layers so you can see exactly where the numbers come from.

By the end, you should be able to explain what distinguishes each of these four architectures, compute the output volume and parameter count of a convolutional or pooling layer, and understand why later architectures moved toward smaller filters, bottleneck layers and residual connections. The lecture closes with a faster survey of more recent research building on and challenging ResNet's design.

Key ideas

• AlexNet (2012): the first CNN to win ImageNet classification by a large margin; eight layers of conv and fully connected layers, using ReLU, dropout, heavy data augmentation, and split across two GPUs due to memory limits.
• Receptive field via small filters: stacking three 3x3 conv layers gives the same effective receptive field as one 7x7 layer, but with fewer parameters and more non-linearities, which is the core design idea behind VGGNet.
• VGGNet (2014): 16 or 19 layers using only 3x3 convolutions throughout; simple and uniform, but memory- and parameter-heavy (about 138 million parameters).
• Inception module (GoogleNet, 2014): applies several filter sizes and pooling in parallel within a module and concatenates the outputs depth-wise, aiming for computational efficiency rather than just depth.
• 1x1 bottleneck convolutions: reduce depth before expensive 3x3 or 5x5 convolutions, cutting the inception module's operation count roughly in half in the example worked through in the lecture.
• Degradation problem: making a plain network deeper (e.g. 56 vs 20 layers) made both training and test error worse, showing the issue is optimization difficulty, not overfitting.
• Residual connections (ResNet, 2015): instead of learning a direct mapping H(x), each block learns a residual F(x) so the output is F(x) + x; this makes it easier for a block to approximate an identity mapping when that's the best choice.
• ResNet (2015): up to 152 layers, using bottleneck blocks in deeper variants, batch normalization, and no dropout; it won every ILSVRC and COCO 2015 competition and beat a reported human error rate on ImageNet classification.
• Efficiency tradeoffs: comparing these architectures by accuracy, operations and memory shows GoogleNet as most efficient, VGG as least efficient, and ResNet as offering the highest accuracy at moderate cost.

Walkthrough

Recap and AlexNet (3:15)

After a brief recap of frameworks and a reminder of LeNet, the lecture works through AlexNet layer by layer. Using the formula (input - filter) / stride + 1, the class computes that an 11x11 conv with 96 filters at stride 4 on a 227x227x3 image produces a 55x55x96 output with about 35K parameters in that layer alone, while the following pooling layer changes spatial size but adds no parameters since pooling has no learned weights. AlexNet's architecture is conv-pool-norm blocks followed by fully connected layers, split across two GPUs because the GTX580 cards used to train it only had 3GB of memory. It won ImageNet 2012 by a wide margin and remained a common base for transfer learning for years afterward.

VGGNet and small filters (15:39)

VGGNet, one of two strong 2014 entrants, uses only 3x3 convolutions stacked with periodic pooling, reaching 16 or 19 layers. The lecture derives why: three stacked 3x3 conv layers give the same effective 7x7 receptive field as a single large filter, but with fewer parameters (3 x 3x3xC^2 versus 7x7xC^2) and more non-linear activations in between. This simple, uniform design reaches 7.3% top-5 error but uses roughly 100MB of memory per image on the forward pass alone and about 138 million parameters, more than double AlexNet's 60 million. Its FC7 layer features are noted as a strong general-purpose feature representation for other tasks.

GoogleNet and the inception module (28:51)

GoogleNet, the other 2014 winner, targets computational efficiency instead of raw depth. Its inception module applies 1x1, 3x3, and 5x5 convolutions and a pooling operation in parallel on the same input, then concatenates the results depth-wise. A worked example shows that naively doing this is expensive (854 million operations for one module), so GoogleNet inserts 1x1 bottleneck convolutions before the expensive filters to project depth down first, cutting the same module to 358 million operations. The full network stacks many inception modules, drops fully connected layers entirely, and adds two auxiliary classifier outputs partway through the network to inject additional gradient signal during training. The result uses only 5 million parameters (12x fewer than AlexNet) at 6.7% top-5 error.

The degradation problem and residual connections (47:26)

Introducing ResNet, the lecture first shows the puzzle that motivated it: a plain 56-layer network has both higher training and test error than a 20-layer version, ruling out overfitting as the cause and pointing instead to an optimization difficulty in very deep plain networks. The proposed fix is a residual block: rather than learning a desired mapping H(x) directly, each block learns a residual F(x) and outputs F(x) + x through an identity shortcut connection. If identity is the best mapping, the block only needs to learn to output zero, which is hypothesized to be easier than learning an exact identity mapping directly.

ResNet architecture and results (58:11)

ResNet stacks these residual blocks, using bottleneck 1x1-3x3-1x1 blocks in networks deeper than 50 layers for efficiency, batch normalization after every convolution, and no dropout. Depths of 34, 50, 101 and 152 layers were tried on ImageNet, with the deepest achieving 3.6% top-5 error, better than a reported human benchmark from a lab member. ResNet swept the ILSVRC and COCO 2015 competitions. The lecture then compares AlexNet, VGG, GoogleNet and ResNet on accuracy, operation count and memory: GoogleNet is most efficient, VGG least, and ResNet delivers the best accuracy at moderate cost.

Later architectures and open questions (1:05:51)

The lecture closes with a faster survey of related work: Network in Network (an early precursor to bottleneck ideas), an improved ResNet block design by the same authors, Wide ResNets (arguing that width, not just depth, drives ResNet's benefit), ResNeXt (adding parallel pathways within each block), Stochastic Depth (randomly dropping layers during training to ease gradient flow), and non-residual alternatives like FractalNet and DenseNet. It ends by mentioning SqueezeNet as an example of designing explicitly for parameter efficiency, achieving AlexNet-level accuracy with far fewer parameters.

Before you watch

• Be comfortable computing convolution and pooling output sizes and parameter counts, since the lecture works several of these by hand.
• Review the earlier lecture on convolutional networks (filters, stride, padding) if the receptive-field calculations feel unfamiliar.
• Knowing the deep learning frameworks covered in Lecture 8 is useful context but not required to follow this lecture's architecture content.

Check your understanding

1. Why does stacking three 3x3 convolutional layers give the same effective receptive field as one 7x7 layer, and why is this more parameter-efficient?
2. What computational problem does the inception module's naive design create, and how do 1x1 bottleneck convolutions address it?
3. What evidence in the lecture rules out overfitting as the explanation for deeper plain networks performing worse than shallower ones?
4. How does a residual block's output relate to its input, and why does this make learning an identity mapping easier?
5. Based on the efficiency comparison near the end, which architecture would you pick if memory and compute were tightly constrained, and which if accuracy were the only priority?