Lecture 18, Variational Inference, Part 2

What this lecture covers

This part derives the core tool of variational inference: a tractable lower bound on log p(x_i), built by approximating the true posterior p(z|x_i) with a simple distribution q_i(z). It shows, via Jensen's inequality, how this bound arises, and then, via KL divergence, why maximizing the bound with respect to q_i tightens it and justifies the expected log likelihood objective introduced in part 1.

By the end, you should be able to derive the evidence lower bound (ELBO), explain the role entropy plays in keeping q_i from collapsing onto a single point, and describe the resulting alternating optimization: maximize the ELBO with respect to q_i to tighten the bound, then with respect to the model parameters to raise the likelihood.

Key ideas

• Jensen's inequality: for a concave function like the logarithm, log(E[y]) >= E[log(y)], which lets the log of an intractable expectation be lower-bounded by a tractable expectation of a log.
• Evidence lower bound (ELBO): log p(x_i) >= E_{q_i(z)}[log p(x_i|z) + log p(z)] + H(q_i), tractable because it only needs samples from q_i(z) and, for distributions like Gaussians, a closed-form entropy.
• Entropy's role: the entropy term keeps q_i(z) spread out rather than collapsing onto the single most likely point, so q_i tends to cover the true posterior rather than just its peak.
• KL divergence: measures how different two distributions are; KL(q_i(z) || p(z|x_i)) is zero exactly when q_i equals the true posterior.
• Exact identity: log p(x_i) = KL(q_i(z) || p(z|x_i)) + ELBO, which shows the ELBO is always a valid lower bound (since KL divergence is non-negative) and that minimizing the KL divergence is equivalent to maximizing the ELBO with respect to q_i.
• Alternating optimization: maximize the ELBO with respect to q_i to tighten the bound, then with respect to the model parameters theta to improve the likelihood, and repeat.
• Scalability problem: fitting a separate q_i (its own mean and variance) for every data point becomes impractical with millions of data points, motivating the move to amortized inference.

Before you watch

• Watch part 1 of this lecture: it introduces latent variable models and the expected log likelihood objective that this part justifies.
• Some familiarity with entropy and basic information theory helps, though the lecture recaps both entropy and KL divergence from first principles.

Check your understanding

1. How does Jensen's inequality turn the intractable log p(x_i) into a tractable lower bound?
2. Why does maximizing the ELBO with respect to q_i alone not just collapse q_i onto the single highest-probability value of z?
3. What does the identity log p(x_i) = KL(q_i(z) || p(z|x_i)) + ELBO tell you about when the ELBO equals the true log likelihood?
4. Why does fitting a separate Gaussian q_i per data point become impractical at scale, and what alternative does the lecture point toward?