Lecture 3: Loss Functions and Optimization

What this lecture covers

Lecture 2 left off with a linear classifier that could compute class scores but gave no way to decide which weight matrix W was actually good. This lecture answers that question in two parts: how to quantify how bad a given W is (loss functions), and how to search for a better one (optimization). It covers the multi-class SVM loss, regularization, the softmax loss, and gradient descent, including its stochastic minibatch version used to train real networks.

This is the third lecture in the course and builds directly on the linear classifier introduced in Lecture 2. After watching, you'll be able to compute both SVM and softmax loss by hand for a small example, explain why regularization is needed, and describe how gradient descent and stochastic gradient descent use the gradient to iteratively improve W.

Key ideas

• Loss function: a function L that takes the predicted scores from a classifier and the true label and returns a number measuring how bad that prediction was; the total loss is the average over the training set.
• Multi-class SVM (hinge) loss: for each incorrect class, it penalizes the classifier if the correct class's score is not at least one point higher than that incorrect class's score, summing these penalties; a perfect score with margin yields zero loss for that example.
• Regularization: an added term R(W), weighted by a hyperparameter lambda, that penalizes model complexity (for example the L2 norm of W) so the classifier prefers simpler solutions that generalize better rather than only fitting the training data.
• Softmax (multinomial logistic regression) loss: converts raw scores into a probability distribution via the softmax function, then computes the loss as the negative log probability assigned to the correct class.
• SVM vs softmax behavior: SVM loss stops caring about an example once its correct score clears the margin; softmax always keeps pushing the correct class's probability toward one, even for examples it already classifies correctly.
• Gradient: the vector of partial derivatives of the loss with respect to each parameter; it points in the direction of steepest increase, so moving in the negative gradient direction decreases the loss.
• Gradient descent: repeatedly compute the loss and gradient, then update W by stepping a small amount (the learning rate) in the negative gradient direction.
• Stochastic gradient descent (SGD): instead of computing the gradient over the full training set (too slow for large datasets), estimate it from a small random minibatch of examples at each step.
• Image features before deep learning: classifiers used to run on hand-designed feature representations, like color histograms, histograms of oriented gradients, or bag-of-visual-words, rather than raw pixels; convolutional networks later replaced this with features learned directly from data.

Walkthrough

From linear classifier scores to a loss function (8:13)

After a recap of the linear classifier from Lecture 2, the lecture explains that having a classifier produce scores is not enough: some settings of W produce better scores than others, and a quantitative measure of "how bad" a given W is needed before it can be automatically improved. This measure is the loss function, and the process of searching for the W that minimizes it is optimization, the two topics the rest of the lecture covers.

The multi-class SVM loss (10:14)

The lecture defines the multi-class SVM (hinge) loss: for a given training example, sum over all incorrect classes the amount by which each incorrect class's score exceeds the correct class's score minus a safety margin of one, using max(0, ...) so scores that already clear the margin contribute nothing. Working through a small three-class, three-image example, the lecture shows how to compute this loss by hand and discusses several properties through Q&A: the loss is unaffected by small changes to already-correct scores, its minimum is zero and its maximum is unbounded, a freshly initialized classifier should show a loss near (number of classes − 1), and the margin of one is an arbitrary but harmless constant since only relative score differences matter.

Regularization (27:28)

The lecture points out a problem: many different W matrices can achieve zero training loss, and a loss function based only on fitting training data can lead to overly complex solutions that fit noise rather than the underlying pattern (illustrated with a wiggly curve that perfectly fits training points but generalizes poorly, versus a simpler line that would generalize better). The fix is to add a regularization term R(W), weighted by hyperparameter lambda, to the loss, encouraging simpler models. The lecture covers L2 regularization (penalizing the norm of W, which spreads influence across many input dimensions), L1 regularization (which favors sparse weight vectors), and briefly mentions elastic net and max-norm regularization.

The softmax loss (38:32)

As an alternative to SVM loss, the lecture introduces the softmax, or multinomial logistic regression, loss. Raw scores are exponentiated and normalized into a probability distribution over classes using the softmax function, and the loss is the negative log of the probability assigned to the true class. Unlike SVM loss, softmax loss gives every score set an interpretation as a probability distribution, and it never stops improving: it keeps pushing the correct class's probability toward one even after a training example is already classified correctly, in contrast to SVM's "good enough" margin-based behavior.

Optimization by gradient descent (49:46)

Framing optimization as walking downhill in a landscape where height equals loss, the lecture rules out random search (tried on CIFAR-10, reaching only about 15% accuracy) as impractical, and instead introduces following the local slope. In one dimension that slope is the derivative; in multiple dimensions it generalizes to the gradient, a vector of partial derivatives that points in the direction of steepest increase. The lecture shows numerically estimating a gradient via finite differences and explains it is far too slow for models with many parameters; in practice, gradients are computed analytically using calculus, with numerical gradients reserved for debugging via "gradient checking." Gradient descent then repeatedly computes the loss and its gradient and steps a small amount, set by a learning rate hyperparameter, in the negative gradient direction.

Stochastic gradient descent (1:03:00)

Because computing the loss over an entire large training set (potentially millions of examples, as with ImageNet) is expensive, the lecture introduces stochastic gradient descent: at each step, sample a small random minibatch of training examples (commonly a power of two like 32, 64, or 128), and use it to compute an estimate of the loss and gradient. This minibatch-based update is the core training algorithm used for essentially all deep neural networks, and the lecture notes that more advanced update rules (such as momentum or Adam) build on this same basic idea and will appear later in the course.

Image features before convolutional networks (1:07:04)

The lecture closes with a look at how image classification worked before deep learning dominated: rather than feeding raw pixels into a linear classifier, practitioners computed hand-designed feature representations, such as color histograms, histograms of oriented gradients (capturing local edge orientations), or bag-of-visual-words codebooks built by clustering image patches, and fed those fixed features into a linear classifier. The lecture frames convolutional neural networks as a natural extension of this same pipeline, except that instead of hand-designing the feature extractor, the network learns the features directly from data during training, a topic the next lecture picks up with neural networks and backpropagation.

Before you watch

• Watch Lecture 2 first; this lecture assumes familiarity with the linear classifier and its f(x, W) = Wx + b form.
• Basic calculus (derivatives) and linear algebra (dot products, vector norms) are needed to follow the gradient and regularization sections.
• Recall from Lecture 2 how train/validation/test splits and hyperparameters work, since the learning rate and regularization strength are both hyperparameters tuned the same way.

Check your understanding

1. How is the multi-class SVM loss computed for a single training example, and why does a "safety margin" of exactly one not actually matter?
2. What problem does regularization solve, and how do L1 and L2 regularization differ in what kind of W they prefer?
3. How does the softmax loss's behavior differ from the SVM loss once a training example is already correctly classified?
4. Why is computing the gradient via finite differences impractical for real neural networks, and what is it still useful for?
5. Why does stochastic gradient descent use small random minibatches instead of computing the gradient over the whole training set?