Lecture 2: Image Classification

What this lecture covers

This lecture asks how a computer can assign a label like "cat" to an image when all it actually receives is a huge grid of pixel numbers. It builds up from the simplest possible approach, nearest neighbor, to the first real learning algorithm used in the course, the linear classifier, and along the way establishes the data-driven approach that underlies the rest of CS231n.

It follows directly from Lecture 1's overview of the field, moving from big-picture history into the first concrete algorithms. After watching, you'll be able to explain why image classification is hard, evaluate a classifier using proper train/validation/test splits, and describe what a linear classifier computes and where it fails.

Key ideas

• The semantic gap: an image is just a grid of RGB pixel values to a computer, with no intrinsic notion of "cat"; recognition means bridging that gap despite viewpoint, lighting, deformation, occlusion, background clutter, and intraclass variation.
• Data-driven approach: rather than hand-coding rules for what a cat looks like, collect a labeled dataset, train a model on it, then use that model to predict labels for new images.
• Nearest neighbor classifier: memorizes all training images at training time, then at test time finds the most similar training image (by some distance metric) and copies its label.
• L1 vs L2 distance: L1 (Manhattan) distance sums absolute pixel differences and depends on the coordinate system; L2 (Euclidean) distance is coordinate-independent; the choice is a hyperparameter.
• k-nearest neighbors (k-NN): instead of using only the single nearest training image, take a vote among the k nearest ones, which smooths out noisy decision boundaries.
• Hyperparameters and validation: values like k or the distance metric can't be learned from training data; the correct way to choose them is to tune on a separate validation set and touch the test set only once, at the end, or use k-fold cross-validation on smaller datasets.
• Why k-NN fails on images: it is slow at test time (the opposite of what's wanted in deployment), pixel distances don't match perceptual similarity, and the "curse of dimensionality" means the number of training examples needed to densely cover pixel space grows exponentially.
• Linear classifier: a parametric model f(x, W) = Wx + b that reduces all training knowledge to a weight matrix W and bias b, computing one score per class as a dot product between the image and a learned per-class template.

Walkthrough

Administrative notes and the CS231n toolchain (0:06)

The lecture opens with logistics: using Piazza for course communication, the release of Assignment 1 (implementing k-nearest neighbor, SVM, Softmax, and a two-layer neural network), the importance of NumPy for vectorized computation, and free Google Cloud credits for running assignments on GPU instances.

Image classification and the semantic gap (5:12)

The lecture defines image classification as assigning an input image one label from a fixed category set, then explains why it's hard: a computer sees only a large grid of numbers, and there's a "semantic gap" between that grid and a concept like "cat." Small changes in viewpoint, lighting, pose, occlusion, background clutter, and the natural variation within a category can all change the pixel grid entirely while the true label stays the same, and there is no simple explicit algorithm (unlike, say, sorting) that solves this directly.

The data-driven approach and nearest neighbor (11:19)

Rather than hand-coding recognition rules, which the lecture shows is brittle and doesn't scale across categories, the data-driven approach collects a large labeled dataset, trains a classifier on it, then predicts on new images. The lecture introduces this with the CIFAR-10 dataset (10 classes, 50,000 training and 10,000 test images) and the simplest possible classifier: nearest neighbor, which memorizes the training set and, at test time, copies the label of the closest training image using a distance metric such as L1 (Manhattan) distance. It also covers the k-nearest neighbors generalization, where voting among the k closest neighbors smooths out noisy decision boundaries, and compares how L1 and L2 distance produce differently shaped decision regions.

Choosing hyperparameters correctly (28:32)

Values like k and the distance metric are hyperparameters that cannot be learned from training data. The lecture walks through why picking hyperparameters based on training accuracy is wrong (k=1 always fits the training set perfectly but generalizes poorly), and why picking them based on test-set performance is also wrong, since it contaminates the test set's role as an unbiased estimate of real-world performance. The correct approach splits data into training, validation, and test sets: tune hyperparameters on the validation set and touch the test set only once, at the very end. For smaller datasets, k-fold cross-validation cycles through which portion of the training data serves as validation, though this is rarely used in deep learning because training is too computationally expensive to repeat many times.

Why nearest neighbor doesn't work well on images (39:48)

The lecture gives three reasons k-NN is almost never used on raw images in practice: it's slow at test time (the reverse of what's needed in deployment, where training can be slow but predictions must be fast), pixel-space distances like L2 don't correspond to how humans perceive image similarity (an example shows three very differently distorted images all having identical L2 distance to the original), and the curse of dimensionality means densely covering high-dimensional pixel space would require an exponentially large number of training examples.

Linear classification (45:53)

The lecture introduces the linear classifier as a parametric model, f(x, W) = Wx + b, that compresses everything learned from training data into a weight matrix W and bias vector b, so the training data itself can be discarded at test time. Each row of W acts as a learned template for one class, and the score for that class is the dot product between the template and the flattened image pixels, plus a bias term. Visualizing these learned templates on CIFAR-10 shows recognizable but blurry class-average images (for example, a horse template that appears to have two heads, since the classifier must average over horses facing both directions with only one template per class). The lecture closes by noting linear classifiers struggle whenever a class isn't linearly separable from the rest, such as parity-style problems or multimodal classes that occupy separate regions of pixel space, and previews that the next lecture will cover how to actually choose the weights W using loss functions and optimization.

Before you watch

• Watch Lecture 1 first for the course's framing of image classification and its history.
• Basic familiarity with vectors, matrices, and dot products will help with the linear classifier section.
• Some exposure to Python and NumPy is useful, since the lecture references vectorized implementations you'll write in Assignment 1.

Check your understanding

1. What is the "semantic gap," and why does it make image classification hard for computers even though it feels effortless to people?
2. Why is it invalid to select hyperparameters based on test-set performance, and what data split should be used instead?
3. What are two specific problems with using a nearest-neighbor classifier directly on image pixels?
4. In a linear classifier f(x, W) = Wx + b, what does each row of W represent, and why does the horse template end up looking like it has two heads?
5. Give an example of a dataset that a linear classifier cannot separate correctly, and explain why.