Lecture 11: Detection and Segmentation

What this lecture covers

Up to this point the course has treated computer vision as image classification: one label per image. This lecture asks a broader question: how do you get a convolutional network to say something about where things are, not just what they are? It works through four increasingly detailed tasks - semantic segmentation, classification plus localization, object detection, and instance segmentation - showing how each one reuses the classification machinery already built up in the course, with new architectural pieces bolted on.

By the end you should be able to describe how a fully convolutional network produces a per-pixel label map, explain what a transpose convolution actually computes, and trace the evolution from R-CNN through Fast R-CNN and Faster R-CNN to single-shot detectors like YOLO and SSD, ending with Mask R-CNN as a model that unifies detection, segmentation, and pose estimation in one network.

Key ideas

• Semantic segmentation: label every pixel of an image with a category, without distinguishing between separate instances of the same class.
• Fully convolutional network: a stack of convolutional layers with no fully-connected layers, trained with a cross-entropy loss averaged over every output pixel.
• Downsampling and upsampling: efficient segmentation networks shrink the feature map with pooling or strided convolution, then grow it back with unpooling or transpose convolution rather than staying at full resolution throughout.
• Transpose convolution: a learnable upsampling operation where each input value weights a copy of the filter that gets placed (and summed where it overlaps) into the output; also called deconvolution, upconvolution, or fractionally strided convolution.
• Classification plus localization: predicts a class label and, separately, four numbers for a bounding box, trained with a weighted multi-task loss combining softmax and a regression loss (L2 or similar).
• Object detection: unlike localization, the number of objects per image is unknown in advance, so the problem cannot be framed as fixed-size regression.
• Region proposals and the R-CNN family: R-CNN, Fast R-CNN, and Faster R-CNN progressively move more of the pipeline (feature sharing, then region proposal generation itself) inside the network to cut cost.
• Single-shot detectors (YOLO, SSD): predict boxes and classes for a grid of base boxes in one forward pass, trading some accuracy for speed against region-based methods.

Walkthrough

Semantic segmentation: from sliding windows to fully convolutional networks (11:16)

The lecture opens the segmentation problem by first describing a naive sliding-window approach: classify the center pixel of every small crop in the image. This is rejected immediately as computationally prohibitive, since neighboring crops share most of their pixels and redundantly repeat convolution work. The alternative is a fully convolutional network that keeps spatial dimensions constant through a stack of convolutions and outputs a tensor of class scores for every pixel at once, trained with cross-entropy loss summed or averaged over all pixels and the batch. Running every layer at full input resolution is itself expensive, which motivates the next section.

Upsampling: unpooling and transpose convolution (19:46)

Practical segmentation networks downsample early (via pooling or strided convolution) and upsample later, keeping most layers at low resolution. The lecture covers three fixed (non-learned) upsampling strategies - nearest neighbor unpooling, bed-of-nails unpooling, and max unpooling, which reuses the positions recorded during the matching max-pooling step - before introducing transpose convolution as a learnable alternative. It walks through the 1D matrix-multiplication view of ordinary convolution and shows that transpose convolution corresponds to multiplying by the transpose of that same weight matrix, which is also mathematically identical to the backward pass of ordinary convolution. A stride-two transpose convolution with a 3x3 kernel can produce checkerboard artifacts because overlapping output regions are summed unevenly; using a 4x4 or 2x2 kernel instead reduces this problem.

Classification plus localization and pose estimation (32:45)

Here the network keeps the classification head but adds a second fully-connected branch predicting four numbers for a single bounding box, assuming exactly one object per image. Training uses a multi-task loss: softmax loss for the class, plus a regression loss (typically L2) between predicted and ground-truth box coordinates, combined with a hyperparameter weight. The lecture notes that this weight is unusually hard to tune by comparing loss values, because it changes the loss itself, so cross-validation should instead track a separate performance metric. The same fixed-output-count idea extends to human pose estimation, where the network regresses the (x, y) coordinates of a fixed number of body joints (for example 14) using a regression loss rather than cross-entropy.

Object detection: region proposals and the R-CNN family (45:40)

Object detection differs from localization because the number of objects per image is unknown, ruling out fixed-size regression. A brute-force sliding-window classifier (with an added "background" category) is again shown to be intractable given the number of possible crop positions, sizes, and aspect ratios. The fix is region proposals: traditional (non-learned) algorithms like Selective Search that generate roughly 2,000 candidate boxes per image. R-CNN classifies each warped proposal crop independently and regresses a correction to its box, but is extremely slow to train and run because every proposal passes through the full network separately. Fast R-CNN fixes this by running the whole image through convolutional layers once and cropping from the resulting feature map using an ROI pooling layer, cutting training time roughly tenfold and shifting the remaining bottleneck to computing the region proposals themselves.

Faster R-CNN, single-shot detectors, and instance segmentation (58:01)

Faster R-CNN removes the external region-proposal bottleneck by adding a region proposal network inside the model that shares the same convolutional features, producing a four-way multi-task loss (proposal objectness, proposal box regression, final classification, final box regression). The lecture then contrasts this region-based family with single-shot detectors, YOLO and SSD, which divide the image into a grid, assign a set of base boxes per cell, and regress offsets and class scores for all of them in a single pass - faster but generally less accurate than Faster R-CNN. It briefly mentions dense captioning as a related idea that swaps the classification head for an RNN language model. The lecture closes with instance segmentation, where Mask R-CNN extends Faster R-CNN with an extra branch that predicts a per-pixel segmentation mask for each detected region, and shows this same architecture can also add a branch for pose estimation, unifying detection, segmentation, and pose in one jointly trained network.

Before you watch

• Be comfortable with the CNN classification pipeline (convolution, pooling, fully-connected output, softmax and cross-entropy loss) from earlier lectures in this course.
• Know what a multi-task loss is and why combining two loss terms with a weighting hyperparameter is different from tuning an ordinary hyperparameter.
• Some familiarity with recurrent networks (image captioning) is useful for the brief dense captioning aside, but not required for the main content.

Check your understanding

1. Why is a naive sliding-window classifier impractical for both semantic segmentation and object detection, and what problem do region proposals or fully convolutional networks solve instead?
2. Explain what a transpose convolution computes and why summing overlapping regions can differ from what plain upsampling would give you.
3. What are the two loss terms in a classification-plus-localization network, and why is their relative weighting harder to tune than a typical hyperparameter?
4. Walk through what changes at each step from R-CNN to Fast R-CNN to Faster R-CNN, and what specifically each step speeds up.
5. How does Mask R-CNN extend Faster R-CNN to perform instance segmentation and pose estimation?