Lecture 13: Expectation-Maximization Algorithms

What this lecture covers

This lecture opens the course's unsupervised learning unit, answering a different question from everything before it: given only unlabeled data x, what structure can you find without any labels y? It starts with k-means clustering as a concrete, intuitive algorithm, then moves to density estimation and anomaly detection, and finally derives the expectation-maximization (EM) algorithm, first informally for fitting a mixture of Gaussians and then rigorously using Jensen's inequality.

The lecture builds directly on the generative learning algorithms covered earlier in the course, particularly Gaussian discriminant analysis, since the mixture of Gaussians model is essentially GDA with the class label replaced by a hidden variable you never observe. After watching, you should be able to run k-means by hand, explain why it converges, set up the mixture of Gaussians model with a latent variable, and follow the E-step and M-step of EM both as a "guess and refit" heuristic and as a lower-bound maximization procedure.

Key ideas

• Unsupervised learning: given a training set of only x_1 through x_m with no labels, the goal is to find interesting structure in the data.
• K-means clustering: alternates between assigning each point to its nearest cluster centroid and moving each centroid to the mean of its assigned points, and this process is guaranteed to converge because it monotonically decreases a non-negative cost function.
• Choosing k: there is no single correct number of clusters; Ng recommends choosing it based on the downstream use of the clusters (for example, how many marketing campaigns a team can realistically run) rather than an automatic criterion.
• Anomaly detection via density estimation: model p(x) from unlabeled data and flag new examples with low probability as potential anomalies, useful when no single feature is unusual but a combination of features is.
• Mixture of Gaussians: models complex, multi-modal densities by assuming each example was generated by one of several Gaussians, selected by a hidden (latent) variable z that is never observed.
• EM as soft k-means: instead of hard-assigning each point to one cluster, the E-step computes a probability (w_ij) that each point belongs to each Gaussian, and the M-step re-estimates the Gaussians using these weights.
• EM as lower-bound maximization: the rigorous derivation constructs a lower bound on the log-likelihood that touches it at the current parameters (E-step), then maximizes that lower bound (M-step), which guarantees the log-likelihood never decreases.
• Jensen's inequality: for a concave function like log, the expectation of the function is less than or equal to the function of the expectation, with equality only when the underlying random variable is constant; this is the tool used to build EM's lower bound.

Walkthrough

From supervised to unsupervised learning and market segmentation (1:54)

The lecture contrasts supervised learning, where you have labeled positive and negative examples, with unsupervised learning, where the training set consists only of unlabeled points x_1 through x_m. Clustering is introduced as the first unsupervised algorithm, with market segmentation as the motivating example: a company with a large user database can run clustering to discover natural groups of customers by attributes like age, education, or region, without having predefined labels for those groups.

K-means clustering, step by step (5:37)

Using a live animation, the lecture walks through k-means: initialize two cluster centroids (commonly by picking k random training examples), then repeat two steps until nothing changes. First, assign each point to the closer of the two centroids. Second, move each centroid to the mean of the points currently assigned to it. The lecture writes this out formally with the assignment variable c_i and centroid variables mu_1 through mu_k, and clarifies that the standard distance measure is the L2 norm, with or without squaring giving the same result.

Why k-means converges, and how to choose k (12:10)

K-means minimizes a cost function equal to the sum of squared distances between each point and its assigned centroid. Because this cost is non-negative and provably decreases on every iteration of the two-step process, it must eventually stop decreasing, which is what convergence means here. In practice, k-means can also get stuck in local minima, so a common fix is to rerun it from many random initializations and keep the run with the lowest cost. Choosing the number of clusters k is treated as inherently ambiguous; rather than using automatic criteria, the lecture recommends choosing k based on what the clusters will be used for.

Density estimation and anomaly detection (16:32)

The lecture shifts to a related but different problem: given unlabeled data such as vibration and heat readings from aircraft engines, model the density p(x) and flag new examples with low probability as anomalies. This approach is described as used in practice for detecting unusual cell-tower network behavior and for computer security monitoring. The key illustration is a case where neither individual feature is out of range, but the specific combination of the two is unusual, which simple per-feature thresholds would miss.

The mixture of Gaussians model (20:40)

To model densities that no single textbook distribution can fit, such as an L-shaped region of probability mass, the lecture introduces a latent variable z that indicates which of several Gaussians generated each example. This closely parallels Gaussian discriminant analysis, except z is multinomial rather than binary, each Gaussian can have its own covariance Sigma_j, and, critically, z is never observed in the training data. If z were known, maximum likelihood estimation would reduce to fitting each Gaussian separately to its labeled points, using formulas nearly identical to GDA.

The EM algorithm's E-step and M-step (31:44)

Since the true values of z are unknown, EM alternates between guessing them and refitting the model. The E-step computes w_ij, the posterior probability that example i came from Gaussian j, using a Bayes' rule calculation that combines the Gaussian density and the multinomial prior over z. The lecture emphasizes that this is a soft assignment: unlike k-means' hard 0-or-1 assignment, each point can be partially assigned to multiple Gaussians, with its weights summing to 1. The M-step then reuses the maximum-likelihood formulas from the "if only we knew z" case, replacing the indicator of z_i = j with the soft weight w_ij.

The rigorous derivation via Jensen's inequality (48:12)

Because the earlier derivation is described as hand-wavy, the lecture builds a formal justification. Jensen's inequality states that for a concave function such as log, E[log(X)] <= log(E[X]), with equality only when X is constant. Applying this to the log-likelihood by introducing an auxiliary distribution Q_i(z_i) over the latent variable produces a lower bound on the log-likelihood as a function of the parameters. Choosing Q_i to make this bound tight at the current parameter value recovers exactly the E-step weights w_ij, and maximizing the resulting lower bound recovers exactly the M-step update, showing that each EM iteration provably does not decrease the log-likelihood and converges to a local optimum.

Before you watch

• Review Gaussian discriminant analysis and Bayes' rule, since the mixture of Gaussians model and the E-step's posterior calculation build directly on them.
• Be comfortable with maximum likelihood estimation and taking derivatives of a log-likelihood, as used in problem set 1's supervised setting.
• A basic notion of convexity or concavity of a function will make the Jensen's inequality derivation easier to follow.

Check your understanding

1. Why is the k-means cost function guaranteed to decrease on every iteration, and why does that guarantee convergence?
2. How does the mixture of Gaussians model extend Gaussian discriminant analysis, and what role does the latent variable z play?
3. What is the difference between the "hard assignment" in k-means and the "soft assignment" computed in EM's E-step?
4. State Jensen's inequality for a concave function, and explain the condition under which it holds with equality.
5. How does choosing Q_i(z_i) to make Jensen's inequality tight lead to the specific formula used in the E-step?