Lecture 14: EM Algorithm and Factor Analysis

What this lecture covers

This lecture continues directly from the EM derivation in the previous session, first showing exactly how the abstract E-step and M-step turn into the concrete update equations for a mixture of Gaussians, and reframing EM as a coordinate ascent algorithm. It then spends most of its time on a new question: how do you model a probability distribution p(x) when your data is high-dimensional but you only have a handful of examples, so few that a standard Gaussian's covariance matrix becomes singular and unusable? The answer is the factor analysis model.

This builds on the previous lecture's mixture of Gaussians and EM derivation, and assumes you are comfortable with multivariate Gaussians, covariance matrices, and the E-step/M-step structure. After watching, you should be able to explain why a naive Gaussian fit fails when the number of examples is smaller than the dimension, set up the factor analysis model with its latent variable z, and follow the key steps used to derive EM for it, including computing Gaussian marginals and conditionals.

Key ideas

• Mapping EM to a concrete model: the abstract E-step (construct a tight lower bound) and M-step (maximize it) become specific formulas once you plug in a particular model, such as w_ij for the mixture of Gaussians E-step and closed-form updates for mu, phi, and Sigma in the M-step.
• EM as coordinate ascent: defining J(theta, Q) as the lower bound, the E-step maximizes J with respect to Q and the M-step maximizes it with respect to theta, alternating one variable at a time while never decreasing the log-likelihood.
• When Gaussian mixtures don't apply: mixture of Gaussians works well when you have far more examples than dimensions; when the number of examples m is close to or smaller than the number of dimensions n, a single Gaussian's maximum likelihood covariance matrix becomes singular and non-invertible.
• Restricting the covariance matrix: forcing the covariance to be diagonal (uncorrelated features) or a scalar multiple of the identity (equal variance, uncorrelated features) avoids the singularity but assumes away the correlations you may actually care about.
• The factor analysis model: assumes a low-dimensional latent Gaussian z (dimension d < n) generates the high-dimensional observation x = mu + Lambda*z + epsilon, where epsilon is independent Gaussian noise with diagonal covariance Psi.
• Geometric intuition: factor analysis models high-dimensional data as lying roughly on a lower-dimensional subspace (a line, plane, or higher-dimensional analogue) with a bit of noise off that subspace.
• Gaussian marginals and conditionals: deriving EM for factor analysis requires knowing that a subvector of a Gaussian is itself Gaussian, and that one subvector conditioned on another is also Gaussian, with mean and covariance given by standard formulas.
• E-step as a continuous distribution: because z is continuous rather than discrete here, the E-step does not store a list of probabilities like w_ij, but instead computes the mean and covariance of a Gaussian distribution over z given x.

Walkthrough

Recapping EM and mapping it to the mixture of Gaussians updates (1:54)

The lecture reviews the general EM framework from the previous session, where the E-step constructs a lower bound tight at the current parameters and the M-step maximizes it. It then works through how this abstract framework produces the specific formulas seen for the mixture of Gaussians: the E-step computes w_ij, the posterior probability of each latent assignment via Bayes' rule, and the M-step takes derivatives of the resulting double sum with respect to phi, mu, and Sigma and sets them to zero to get closed-form updates.

EM as coordinate ascent (16:06)

Defining a function J(theta, Q) equal to the lower bound used in the E-step, the lecture shows that L(theta) >= J(theta, Q) for any Q, and that the E-step is exactly maximizing J with respect to Q while the M-step maximizes J with respect to theta. This reframes EM as coordinate ascent on J, alternating between the two variables, and gives a practical way to monitor convergence: plot the value of J or the log-likelihood across iterations and watch for it to plateau.

Why mixture of Gaussians fails on small, high-dimensional data (17:23)

The lecture introduces factor analysis by contrast: mixture of Gaussians works well when you have many more examples than dimensions, but fails when dimensions exceed or roughly match the number of examples. The motivating example is a Stanford building with around 100 temperature sensors but only around 30 days of readings, where the goal is to model the joint distribution of sensor readings to detect anomalies such as an open window. A second example, from the model's psychometric origins, involves scoring 100 survey questions from only 30 respondents.

The singular covariance problem and its partial fixes (24:03)

When the number of examples is at or below the number of dimensions, the maximum likelihood covariance matrix for a single Gaussian becomes singular, meaning its determinant is zero and it cannot be inverted, so the Gaussian density formula breaks down. The lecture illustrates this with a two-point, two-dimensional example where the fitted Gaussian collapses onto an infinitely thin line. Two workarounds are shown and both are treated as inadequate: forcing the covariance matrix to be diagonal removes off-diagonal correlations entirely, and forcing it to be a scalar multiple of the identity additionally forces every feature to have equal variance. Both avoid the singularity but assume features are uncorrelated, which is usually unrealistic and is presented as a stepping stone toward factor analysis rather than a real solution.

Defining the factor analysis model (40:34)

Factor analysis introduces a continuous latent variable z, drawn from a standard Gaussian in a lower dimension d, and models the observed data as x = mu + Lambda*z + epsilon, where Lambda is an n by d matrix and epsilon is Gaussian noise with a diagonal covariance Psi. The lecture works through low-dimensional geometric examples, mapping a one-dimensional z into two dimensions and a two-dimensional z into three dimensions, to show how this produces data clustered near a line or plane with added noise. This gives the intuition for how high-dimensional data (say, 100 sensor readings) can be explained by a small number of underlying driving factors plus independent per-sensor noise.

Deriving EM for factor analysis (42:29)

Because there is no closed-form maximum likelihood solution for factor analysis, the lecture derives EM for it, describing this as one of the trickier derivations in the course. It first establishes that a subvector of a jointly Gaussian vector is itself Gaussian (the marginal) and that one part conditioned on another is also Gaussian with standard formulas for the conditional mean and covariance. Using these, the joint distribution of z and x is shown to be Gaussian, letting the E-step represent the posterior over z given x as a Gaussian with a specific mean and covariance rather than a discrete list of probabilities. The lecture also highlights a general trick for the M-step: when an integral against a Gaussian density has a log in front, it usually simplifies to a quadratic function, and when it is really computing an expectation, it can often be replaced directly by an already-computed mean rather than expanded as an integral.

Before you watch

• Watch the previous lecture on the EM algorithm and mixture of Gaussians, since this lecture builds directly on its E-step and M-step derivation.
• Review the properties of the multivariate Gaussian distribution, particularly marginal and conditional distributions, since these are used as building blocks for the factor analysis derivation.
• Be comfortable with matrix notation for covariance matrices, including what it means for a matrix to be singular or non-invertible.

Check your understanding

1. Why does the maximum likelihood covariance matrix for a single Gaussian become singular when the number of examples is smaller than the number of dimensions?
2. In what sense is EM a form of coordinate ascent, and what two things does it alternate optimizing?
3. What do the parameters mu, Lambda, and Psi represent in the factor analysis model x = mu + Lambda*z + epsilon?
4. Why does restricting the covariance matrix to be diagonal or a scalar multiple of the identity solve the singularity problem but still produce an unsatisfying model?
5. Why is the E-step for factor analysis represented as a Gaussian distribution over z rather than a list of probabilities like in the mixture of Gaussians?