Lecture 7: Kernels and the Support Vector Machine

What this lecture covers

The previous lecture defined the optimal margin classifier and posed maximizing the geometric margin as an optimization problem. This lecture completes that derivation, introduces the representer theorem's key idea that the classifier's weight vector can be written as a combination of training examples, and shows how rewriting the whole algorithm in terms of inner products enables the kernel trick. With kernels, an SVM can effectively operate in feature spaces with millions or infinitely many dimensions while only ever computing in the dimension of the original input.

After watching, you can explain why the SVM optimization can be rewritten in terms of inner products, state the kernel trick's four-step recipe, compute a simple polynomial kernel efficiently, and describe how the soft-margin SVM handles data that isn't perfectly separable.

Key ideas

• Optimal margin classifier: chooses w and b to maximize the worst-case geometric margin across training examples, formulated as minimizing (1/2)||w||^2 subject to each example having functional margin at least 1.
• Representer theorem: at the optimum, the weight vector w can always be written as a linear combination of the training examples, w = sum_i alpha_i y_i x_i, which lets the algorithm be re-expressed entirely in terms of the training data.
• Inner product formulation: once w is replaced by this sum, both the optimization objective and the prediction rule depend on feature vectors only through inner products x_i . x_j, never through the individual features themselves.
• Dual optimization problem: a further simplification, obtained through convex optimization theory, expresses the SVM training problem purely in terms of the parameters alpha_i.
• Kernel function: K(x, z) = phi(x)^T phi(z), a function that computes the inner product of two examples' high-dimensional feature mappings without ever forming those mappings explicitly.
• Kernel trick recipe: write an algorithm in terms of inner products, choose a feature mapping phi, find an efficient way to compute K(x,z) = phi(x)^T phi(z), then replace every inner product in the algorithm with K(x,z).
• Mercer's theorem: a function K is a valid kernel (corresponds to some feature mapping phi) if and only if its kernel matrix over any finite set of points is positive semi-definite.
• L1 soft-margin SVM: relaxes the requirement that every example have functional margin at least 1, adding a penalty for violations, which makes the classifier robust to outliers and usable when data isn't perfectly separable.

Walkthrough

Deriving the optimal margin classifier's optimization problem (2:47)

The lecture fills in the derivation skipped last time: maximizing the geometric margin gamma, subject to every example having geometric margin at least gamma, is rewritten by noting that scaling w and b doesn't change the decision boundary. Fixing the scale so that ||w|| = 1/gamma turns the problem into minimizing (1/2)||w||^2 subject to every example having functional margin at least 1, a convex optimization problem with a unique solution.

The representer theorem: w as a combination of training examples (11:58)

To eventually work with extremely high-dimensional or infinite-dimensional features, the lecture introduces the assumption, proven formally in the lecture notes as the representer theorem, that the optimal w can always be written as w = sum_i alpha_i y_i x_i. Two intuitions support this: gradient descent, starting from w=0, only ever adds multiples of training examples to w; and geometrically, w is always perpendicular to the decision boundary and lies in the span of the training examples.

Rewriting the optimization problem with inner products (the dual problem) (26:31)

Substituting the representer-theorem expression for w into the optimization objective turns ||w||^2 into a sum involving only the inner products x_i . x_j between training examples, and the constraints turn into expressions involving x_i . x. A further, more involved simplification using convex optimization theory produces the dual optimization problem, expressed purely in terms of the alpha parameters. Predictions on a new example also reduce to a sum of inner products between the new example and the training examples, weighted by the alphas.

The kernel trick (28:48)

Because the entire algorithm, training and prediction, depends on feature vectors only through inner products, the lecture presents the kernel trick as a four-step recipe: express the algorithm in terms of inner products, choose a high-dimensional feature mapping phi, find an efficient way to compute the kernel function K(x,z) = phi(x)^T phi(z), and substitute K(x,z) everywhere an inner product appears. This avoids ever computing phi(x) explicitly, which matters because phi(x) can have hundreds of thousands, millions, or infinitely many dimensions.

Computing kernels efficiently and testing validity (34:40)

Using a feature mapping of all pairwise products of input features, the lecture proves that K(x,z) = (x^T z)^2 computes the same result as explicitly forming and taking the inner product of the mapped vectors, but in linear rather than quadratic time in the input dimension. Variants like (x^T z + c)^d correspond to feature mappings containing all monomials up to degree d, still computable in linear time. Not every similarity function is a valid kernel: Mercer's theorem states that a function K is a valid kernel if and only if its kernel matrix over any finite set of points is positive semi-definite, which the lecture partially proves by showing this condition is necessary.

The L1 soft-margin SVM for non-separable data (1:04:07)

The algorithm so far assumes the data is linearly separable, even after mapping to a high-dimensional space, and always seeks a decision boundary with zero training error. The L1 soft-margin SVM relaxes the functional margin constraint from 1 to 1 - c_i, with each c_i >= 0, and adds a penalty term for large c_i to the objective. This makes the classifier robust to a small number of outliers or noisy points that would otherwise force a poorly shaped decision boundary, at the cost of introducing a parameter C that trades off margin size against training accuracy.

Kernel examples: digit and protein classification (1:14:13)

The lecture closes with concrete applications: SVMs with a polynomial or Gaussian kernel performed well on the MNIST handwritten digit classification benchmark, using raw pixel intensities as features, and were long considered state of the art before deep learning surpassed them. A protein sequence classification example shows how a specialized kernel, computed efficiently with a dynamic-programming algorithm related to string-matching methods, can measure similarity between variable-length amino acid sequences without ever building the underlying high-dimensional feature vectors explicitly.

Before you watch

• Review functional and geometric margin and the optimal margin classifier's optimization problem from Lecture 6, since this lecture builds directly on that formulation.
• Be comfortable with vector inner products and basic matrix notation, since the kernel trick is defined entirely in those terms.
• Familiarity with convex optimization is helpful but not required; the lecture treats the detailed derivations as reference material in the lecture notes.

Check your understanding

1. Why does substituting the representer-theorem expression for w let the entire SVM optimization problem be written using only inner products between training examples?
2. What is the four-step recipe for applying the kernel trick to a learning algorithm?
3. Why does computing K(x,z) = (x^T z)^2 take linear time in the input dimension, even though it corresponds to an inner product in a quadratic-dimensional feature space?
4. What problem does the L1 soft-margin SVM solve that the basic optimal margin classifier cannot handle well?