Lecture 18, Variational Inference, Part 1

What this lecture covers

This lecture shifts away from reinforcement learning algorithms to lay the groundwork for variational inference, a tool the course will connect back to model-based RL, inverse RL, and exploration in later lectures. It explains what a latent variable model is, why mixture models and mixture density networks are examples of them, and how they let a complex distribution be built as a product or sum of simple distributions.

By the end, you should be able to explain why maximum-likelihood training of a latent variable model requires integrating out the latent variable, why that integral is intractable for continuous z, and why the lecture turns instead to an "expected log likelihood" objective that can be estimated with samples once the posterior p(z|x) is available.

Key ideas

• Probabilistic model: any model representing a distribution p(x), or a conditional distribution such as p(y|x) (a policy is an example of the latter).
• Latent variable model: a model with variables that are neither the evidence nor the query, so they must be integrated out to compute the probability of interest.
• Mixture model example: data with unlabeled clusters can be modeled as p(x) = sum_z p(x|z) p(z), where the discrete z indicates cluster identity.
• Simple times simple can be complex: both p(z) and p(x|z) can be simple distributions (such as Gaussians), yet integrating out z can produce a very complex p(x).
• Generative vs. latent variable models: a generative model produces x; a latent variable model has latent variables; the two overlap but are not the same, and complex generative models are often easiest to represent as latent variable models.
• Maximum likelihood objective: fitting theta to maximize the average log probability of the data, which becomes intractable when it requires integrating over a continuous z.
• Expected log likelihood: an alternative objective that averages log p_theta(x_i, z) under the posterior p(z|x_i), which is tractable because expectations, unlike the log of an integral, can be estimated by sampling.

Before you watch

• Review mixture density networks from the imitation learning lecture earlier in the course, since they are used here as a concrete example of a conditional latent variable model.
• Recall the model-based RL lectures on learning latent state models from images, which this lecture references as another use of latent variable structure.

Check your understanding

1. What makes a model a "latent variable model" as opposed to just a conditional or unconditional probabilistic model?
2. Why can a mixture of two simple distributions, p(z) and p(x|z), represent a p(x) that is far more complex than either one alone?
3. Why is the direct maximum-likelihood objective for a latent variable model often intractable to optimize, especially when z is continuous?
4. What property of expectations makes the expected log likelihood objective tractable to estimate, when the direct log-likelihood objective is not?