Lecture 6: Laplace Smoothing and Support Vector Machines

What this lecture covers

Monday's lecture left Naive Bayes with a flaw: any word never seen in a training example forces a probability estimate of zero, which can crash the whole classifier. This lecture fixes that with Laplace smoothing, introduces a second Naive Bayes variant built for text (the multinomial event model), and gives practical advice on how to start a machine learning project. It then turns to a new classifier, the support vector machine, and builds up the two core concepts, functional margin and geometric margin, that the SVM optimization problem is built from.

After watching, you can explain why Laplace smoothing prevents zero-probability estimates, describe the difference between the multivariate Bernoulli and multinomial event models for text, and define functional and geometric margin well enough to follow how the optimal margin classifier chooses a decision boundary.

Key ideas

• Laplace smoothing: add 1 to each outcome's count and add the number of possible outcomes to the denominator, so no probability estimate is ever exactly 0 or 1, even for events never observed in training.
• Multivariate Bernoulli event model: the original Naive Bayes representation, where each email becomes a fixed-length binary vector marking which dictionary words appear, ignoring word order and repeat counts.
• Multinomial event model: an alternative text representation where the feature vector has one entry per word position in the email, so it varies in length with the email and can better use word-frequency information.
• Quick and dirty first, then iterate: build a simple algorithm first, run error analysis on what it gets wrong, and let that evidence guide where to invest further effort, rather than guessing which improvement matters most in advance.
• Functional margin: a measure of how confidently and correctly a linear classifier labels one example, defined for the SVM using labels of +1/-1 rather than 0/1.
• Geometric margin: the actual Euclidean distance from a training example to the decision boundary; equal to the functional margin divided by the norm of the weight vector w.
• Optimal margin classifier: the algorithm that, for linearly separable data, chooses the hyperplane maximizing the worst-case geometric margin across all training examples.
• Reformulated optimization: maximizing the geometric margin subject to a per-example constraint can be rewritten as an equivalent convex problem: minimize the norm of w subject to the same margin constraints.

Walkthrough

Laplace smoothing fixes zero-probability estimates (0:03)

The lecture recaps the Naive Bayes model from Monday, then shows where it breaks: if a word like "NIPS" never appears in any training email, its maximum-likelihood probability estimate is 0 for both spam and non-spam, and multiplying that zero into a later prediction wipes out the whole probability. Using a Stanford football team's losing streak as an example, the lecture motivates Laplace smoothing: instead of estimating a win probability as 0/4, add 1 to the numerator and add the number of possible outcomes (2, for win/loss) to the denominator. Applied to Naive Bayes, this means adding 1 to each word-count numerator and 2 to each denominator, which keeps every estimate strictly between 0 and 1.

The multinomial event model for text (17:29)

The lecture generalizes Naive Bayes to multinomial (not just binary) features, using a house-price bucket example, before introducing a representation designed specifically for text: instead of a fixed 10,000-dimensional binary vector, each email is represented as a variable-length vector whose entries are word indices, one per word in the email. This multinomial event model can capture that a word appeared multiple times, something the original binary (multivariate Bernoulli) representation discards. Its maximum-likelihood parameters are estimated by counting how often each word occurs across all words in spam versus non-spam emails, with Laplace smoothing again adding 1 to the numerator and the dictionary size to the denominator.

Advice for applying machine learning algorithms (35:30)

Naive Bayes is rarely the most accurate algorithm, but its speed and simplicity make it a good first attempt. The lecture recommends building a quick, simple version of a project first, rather than committing weeks to a single sophisticated idea, and using error analysis on real misclassified examples to decide where further effort is actually worth spending. A spam filter example illustrates this: deliberately misspelled words, spoofed headers, and fetched URLs are all plausible improvements, but only examining what the current classifier gets wrong reveals which is worth pursuing.

Introducing support vector machines and non-linear boundaries (43:40)

The lecture motivates SVMs with a dataset that needs a non-linear decision boundary. One option is to hand-engineer polynomial features for logistic regression, but choosing which powers and products to add is not obvious. SVMs, by contrast, can map features into a much higher-dimensional space, later shown to potentially be infinite-dimensional through kernels, and then fit a linear classifier there. A practical draw of SVMs is also mentioned: mature software packages make them close to a turnkey algorithm with few parameters to tune, unlike gradient descent's learning rate.

Functional margin (53:58)

Using logistic regression's threshold at theta^T x = 0 as a starting point, the lecture defines the functional margin of a hyperplane, parameterized here by w and b with labels of +1/-1, with respect to one example as y_i * (w^T x_i + b). A large positive functional margin means the classifier is both correct and confident on that example. The functional margin of an entire training set is defined as the minimum functional margin across all examples, a worst-case measure. The lecture also flags that this quantity can be inflated artificially just by scaling w and b, without changing the actual decision boundary.

Geometric margin (58:03)

To avoid that scaling issue, the lecture defines the geometric margin as the actual Euclidean distance from a training example to the decision boundary, computed as (w^T x + b) / ||w||. This normalizes away the scaling problem in the functional margin, and the two are related by geometric margin = functional margin / ||w||. Like the functional margin, the geometric margin of a training set is defined as the worst-case value over all examples, and a larger geometric margin corresponds to a decision boundary with more physical separation from the data.

The optimal margin classifier's optimization problem (1:17:23)

The optimal margin classifier chooses w and b to maximize the training set's geometric margin, assuming the data is linearly separable. The lecture sketches how this is posed as an optimization problem, maximizing gamma subject to every example having geometric margin at least gamma, and notes that this non-convex form can be rewritten into an equivalent convex problem: minimize the norm of w subject to the same per-example margin constraints. This convex form is solvable with standard numerical optimization packages, and forms the foundation the full SVM will build on once kernels are introduced.

Before you watch

• Review the Naive Bayes model and Gaussian discriminant analysis from Lecture 5, since this lecture builds directly on both.
• Recall logistic regression's decision rule (predicting based on the sign of theta^T x), which is used as the starting point for defining the SVM's margin.
• Basic comfort with vector norms and dot products will help with the functional and geometric margin definitions.

Check your understanding

1. Why does the maximum-likelihood estimate for Naive Bayes break down for a word that never appeared in training, and how does Laplace smoothing fix it?
2. What is the key difference between the multivariate Bernoulli and multinomial event models for representing text, and what information does the multinomial model preserve that the other discards?
3. How does the geometric margin differ from the functional margin, and why is the geometric margin not affected by rescaling w and b?
4. What optimization problem does the optimal margin classifier solve, and what assumption about the training data does it currently rely on?