Lecture 5: GDA and Naive Bayes

What this lecture covers

Earlier lectures built discriminative algorithms that learn P(y|x) directly, such as logistic regression. This lecture introduces a different family: generative learning algorithms, which model P(x|y) and P(y) for each class separately and then combine them with Bayes' rule to classify a new example. It works through Gaussian discriminant analysis (GDA), a generative model for continuous features, and Naive Bayes, a generative model for discrete features such as the words in an email.

After watching, you can write down the GDA and Naive Bayes models, derive their maximum-likelihood parameter estimates, and explain when a generative model like GDA is likely to outperform a discriminative model like logistic regression, and when it is not.

Key ideas

• Discriminative vs. generative: a discriminative algorithm learns P(y|x) directly; a generative algorithm learns P(x|y) for each class plus the class prior P(y), then uses Bayes' rule at test time.
• Class prior: P(y), the probability of a class before seeing any features (for example, the base rate of malignant tumors).
• Gaussian discriminant analysis (GDA): assumes x|y=0 and x|y=1 are each multivariate Gaussian, sharing one covariance matrix Sigma but with different means mu0 and mu1.
• Multivariate Gaussian: a distribution over vector-valued random variables, parameterized by a mean vector and a covariance matrix; the covariance matrix's shape and off-diagonal values control how the density is stretched, tilted, and correlated.
• Maximum likelihood fitting: GDA's parameters are estimated in closed form: phi is the fraction of positive examples, mu0/mu1 are the mean feature vectors of each class, and Sigma is a pooled covariance.
• GDA implies a linear decision boundary: with a shared covariance matrix, GDA's posterior P(y=1|x) takes the same sigmoid shape logistic regression produces, but the two algorithms can end up with different boundaries.
• Stronger vs. weaker assumptions: GDA assumes Gaussian features, a strong assumption that helps when it is roughly true and data is scarce; logistic regression makes a weaker assumption and is more robust when the data's true distribution is unknown.
• Naive Bayes conditional independence assumption: for discrete features like the words in an email, Naive Bayes assumes each feature is independent of the others given the class label, which collapses an intractable number of parameters into one per word per class.

Walkthrough

Discriminative vs. generative learning algorithms (0:03)

The lecture opens by contrasting the two philosophies using a tumor classification example. A discriminative algorithm like logistic regression searches directly for a decision boundary that separates the positive and negative examples. A generative algorithm instead looks at each class in isolation, builds a model of what that class's features look like, and classifies a new example by comparing it against each class model. Formally, discriminative algorithms learn P(y|x), while generative algorithms learn P(x|y) and P(y), then apply Bayes' rule to recover P(y|x).

The multivariate Gaussian distribution (9:19)

Before defining GDA, the lecture reviews the multivariate Gaussian as the generalization of the familiar bell curve to vector-valued random variables, with mean vector mu and covariance matrix Sigma. Through a series of plots, it shows how shrinking or enlarging the covariance matrix narrows or widens the density, and how changing the off-diagonal entries tilts the contours to reflect positive or negative correlation between dimensions, while moving mu shifts the bump's location.

The GDA model and Bayes' rule (18:49)

GDA models x|y=0 and x|y=1 as Gaussians sharing one covariance matrix Sigma but with separate means mu0 and mu1, and models y as a Bernoulli variable with parameter phi. Given these four parameters, a new example's class can be predicted by plugging its features into Bayes' rule and comparing the resulting probabilities.

Fitting GDA parameters by maximum likelihood (22:58)

Unlike discriminative models, which maximize the conditional likelihood P(y|x), GDA is fit by maximizing the joint likelihood P(x,y) over the training set. Taking logs and derivatives yields closed-form estimates: phi is the fraction of training examples with y=1, mu0 and mu1 are the average feature vectors of each class, and Sigma is estimated from both classes together using their respective means.

GDA vs. logistic regression decision boundaries (36:47)

Running both algorithms on the same two-feature dataset shows logistic regression's line moving iteratively via gradient descent, while GDA fits two Gaussian bumps to the classes and derives its boundary from Bayes' rule. The lecture demonstrates that because the two Gaussians share a covariance matrix, GDA's boundary is also linear, but the two algorithms generally produce different lines from the same data. Sweeping P(y=1|x) across values of x for a fitted GDA model traces out a sigmoid curve, the same functional form logistic regression assumes directly.

Choosing between generative and discriminative assumptions (48:05)

The lecture compares the assumptions each algorithm makes: GDA's Gaussian assumption implies a logistic form for P(y=1|x), but the reverse is not true, so GDA makes a strictly stronger assumption than logistic regression. Stronger, correct assumptions help most when training data is limited, because they inject extra knowledge the data alone would take more examples to learn; if the Gaussian assumption is wrong, GDA suffers. The same logistic form also follows from other exponential-family assumptions, such as Poisson features, which is part of why logistic regression is robust across many real data distributions. With very large datasets, the discussion notes, the trend favors discriminative models that assume less and let the data do more of the work; with small datasets, encoding correct assumptions or domain knowledge still matters, and GDA also remains attractive because its parameters are cheap to compute without an iterative optimization.

Naive Bayes for spam classification (1:03:34)

The lecture turns to a second generative model for discrete data, using email spam detection as the running example. Each email is mapped to a binary feature vector indicating whether each word from a fixed vocabulary appears in it. Modeling the joint distribution of thousands of binary features directly would require an intractable number of parameters, so Naive Bayes assumes each word's presence is conditionally independent of the others given the class label. This reduces the model to one parameter per word per class plus a class prior, all estimated by simple counting: the fraction of spam emails containing each word, and the fraction of emails overall that are spam.

Before you watch

• Review logistic regression and the generalized linear model framework from the previous lecture, since this lecture repeatedly contrasts GDA against them.
• Be comfortable with Bayes' rule and basic probability notation, including conditional probability.
• A prior exposure to the univariate Gaussian distribution helps; the lecture builds the multivariate case from there.

Check your understanding

1. Why does GDA use two different means but a single shared covariance matrix for its two classes, and what does that assumption imply about the decision boundary?
2. How do the maximum-likelihood estimates for mu0 and mu1 relate to simply averaging features within each class?
3. Under what conditions does the lecture suggest GDA will outperform logistic regression, and when does logistic regression tend to win instead?
4. What is the Naive Bayes conditional independence assumption, and why is it needed to make the spam classification model tractable?