Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 73 of 99 · 20:12

Lecture 18, Variational Inference, Part 1

CS 285: Lecture 18, Variational Inference, Part 1 on YouTube

Study guide

What this lecture covers

This lecture shifts away from reinforcement learning algorithms to lay the groundwork for variational inference, a tool the course will connect back to model-based RL, inverse RL, and exploration in later lectures. It explains what a latent variable model is, why mixture models and mixture density networks are examples of them, and how they let a complex distribution be built as a product or sum of simple distributions.

By the end, you should be able to explain why maximum-likelihood training of a latent variable model requires integrating out the latent variable, why that integral is intractable for continuous z, and why the lecture turns instead to an "expected log likelihood" objective that can be estimated with samples once the posterior p(z|x) is available.

Key ideas

  • Probabilistic model: any model representing a distribution p(x), or a conditional distribution such as p(y|x) (a policy is an example of the latter).
  • Latent variable model: a model with variables that are neither the evidence nor the query, so they must be integrated out to compute the probability of interest.
  • Mixture model example: data with unlabeled clusters can be modeled as p(x) = sum_z p(x|z) p(z), where the discrete z indicates cluster identity.
  • Simple times simple can be complex: both p(z) and p(x|z) can be simple distributions (such as Gaussians), yet integrating out z can produce a very complex p(x).
  • Generative vs. latent variable models: a generative model produces x; a latent variable model has latent variables; the two overlap but are not the same, and complex generative models are often easiest to represent as latent variable models.
  • Maximum likelihood objective: fitting theta to maximize the average log probability of the data, which becomes intractable when it requires integrating over a continuous z.
  • Expected log likelihood: an alternative objective that averages log p_theta(x_i, z) under the posterior p(z|x_i), which is tractable because expectations, unlike the log of an integral, can be estimated by sampling.

Before you watch

  • Review mixture density networks from the imitation learning lecture earlier in the course, since they are used here as a concrete example of a conditional latent variable model.
  • Recall the model-based RL lectures on learning latent state models from images, which this lecture references as another use of latent variable structure.

Check your understanding

  1. What makes a model a "latent variable model" as opposed to just a conditional or unconditional probabilistic model?
  2. Why can a mixture of two simple distributions, p(z) and p(x|z), represent a p(x) that is far more complex than either one alone?
  3. Why is the direct maximum-likelihood objective for a latent variable model often intractable to optimize, especially when z is continuous?
  4. What property of expectations makes the expected log likelihood objective tractable to estimate, when the direct log-likelihood objective is not?

Vocabulary

variational inference (noun)
A set of methods for approximating hard-to-compute probability distributions with simpler ones.
This lecture lays the groundwork for variational inference.
probabilistic model (noun)
A model that represents outcomes using probabilities instead of fixed values.
A policy is an example of a probabilistic model.
latent variable (noun)
A hidden variable in a model that is not directly observed.
The cluster identity is a latent variable in the mixture model.
latent variable model (noun)
A model that includes hidden variables which must be integrated out to get a prediction.
A mixture density network is a latent variable model.
integrate out (phrasal verb)
To remove a variable from a formula by summing or averaging over all its possible values.
The latent variable must be integrated out to get the final probability.
mixture model (noun)
A model that combines several simple distributions to represent more complex data.
Unlabeled clusters can be modeled with a mixture model.
cluster (noun)
A group of similar data points.
Each cluster corresponds to one value of the latent variable.
Gaussian (adjective)
Following the bell-shaped normal probability distribution.
Both distributions in the mixture can be Gaussian.
generative model (noun)
A model that can produce new data samples similar to its training data.
A generative model produces realistic new images.
maximum likelihood (noun)
A method that picks model parameters to make the observed data as probable as possible.
Training uses the maximum likelihood objective.
intractable (adjective)
Too difficult or slow to compute exactly, even in principle.
The exact integral becomes intractable for continuous variables.
posterior (noun)
The updated probability of a hidden variable after seeing the data.
The expected log likelihood uses the posterior over the latent variable.
expectation (noun)
The average value of a quantity, weighted by how likely each outcome is.
Expectations can be estimated by sampling.
conditional (adjective)
Depending on the value of another variable.
A policy is a conditional distribution over actions given a state.
query (noun)
The specific variable or quantity a model is asked to predict.
In a probabilistic model, the query is what we want the output to be.
evidence (noun)
The observed data used as input to a probabilistic model.
x is the evidence in the model p(y|x).
discrete (adjective)
Made up of separate, distinct values rather than a smooth range.
The cluster identity z is a discrete latent variable.
continuous (adjective)
Able to take any value within a range, not just separate distinct ones.
The integral is intractable when z is continuous.
groundwork (noun)
The basic preparation needed before building something more complex.
This lecture lays the groundwork for variational inference.
represent (verb)
To stand for or express something in a particular form.
A complex distribution can be represented as a mixture of simple ones.
overlap (verb)
To share some parts in common with something else.
Generative models and latent variable models overlap but are not the same.
average (verb)
To compute the typical value across many examples.
Maximum likelihood averages the log probability of the data.
estimate (verb)
To calculate an approximate value using limited information.
Expectations can be estimated by sampling instead of computed exactly.
tractable (adjective)
Simple enough to compute or solve in practice.
The expected log likelihood objective is tractable to estimate.
unlabeled (adjective)
Not marked with a category or class.
Unlabeled clusters can be modeled with a mixture model.

Chapters

← Lecture 17, Part 2: RL Theory · Lecture 18, Variational Inference, Part 2 →