Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 76 of 99 · 25:29

Lecture 18, Variational Inference, Part 4

Study guide

What this lecture covers

This final part of the lecture applies the amortized variational inference machinery from earlier parts to concrete architectures. It walks through the variational autoencoder (VAE), explains why its representations tend to disentangle underlying factors of variation, and shows how it can be used for representation learning in RL. It then extends the idea to conditional VAEs for multimodal policies, and finally to sequence VAEs used as state space models for partially observed environments.

After watching, you should be able to describe the VAE training objective and computation graph, explain intuitively why sampling from the unit-variance prior produces valid decoded outputs, and describe how the encoder, decoder, and prior are restructured for conditional and sequential settings.

Key ideas

Variational autoencoder (VAE): an encoder network q_phi(z|x) outputs a mean and variance for z, a decoder network p_theta(x|z) reconstructs x; both are trained together to maximize the evidence lower bound using the reparameterization trick.
Why VAE samples look valid: the KL term pushes the encoder's output distribution toward the unit-variance prior, and the encoder is incentivized to use the latent space efficiently, so most points sampled from the prior decode into plausible outputs.
Representation learning for RL: training a VAE on states (such as Atari frames) and feeding the learned z into a Q-function instead of raw pixels can give a more useful, disentangled state representation.
Disentanglement: because VAE latent dimensions are encouraged to be independent, they tend to separate underlying factors of variation (position, orientation, lighting) better than raw pixels do.
Conditional VAE: extends the VAE by giving both encoder and decoder an additional input x, modeling p(y|x,z), useful for representing multimodal policies in imitation learning.
Sequence VAE (state space model): treats an entire trajectory of latent states z_1...z_T as the latent variable, with a learned, structured prior p(z_{t+1}|z_t,a), a per-timestep decoder p(o_t|z_t), and an encoder that infers z_t from the history of observations.
Applications: sequence VAEs have been used to learn state space models for planning (for example, Embed to Control) and to provide representations for actor-critic RL on image observations (for example, Stochastic Latent Actor-Critic).

Before you watch

Watch parts 1 through 3 of this lecture, which derive the evidence lower bound and the reparameterization trick used throughout this part.
Recall the model-based RL lectures on learning latent state models from images, since the sequence VAE revisits that material with the variational inference framework now available.

Check your understanding

Why does the KL-divergence term in the VAE objective encourage the encoder to produce latent codes that resemble samples from the prior?
In what sense does a VAE trained on images tend to disentangle the underlying factors of variation, and why might that help a downstream Q-function?
What changes in the encoder, decoder, and prior when moving from a standard VAE to a conditional VAE for multimodal policies?
Why must the encoder in a sequence VAE depend on the history of observations, while the decoder can be independent per timestep?

Vocabulary

variational autoencoder (noun): A model with an encoder and decoder trained together to compress and reconstruct data.
A variational autoencoder learns a compact representation of images.
reconstruct (verb): To rebuild the original input from a compressed representation.
The decoder tries to reconstruct the original image.
disentangle (verb): To separate mixed factors so each one is represented independently.
A VAE's latent space can disentangle position from lighting.
representation learning (noun): Training a model to produce useful, compact features from raw data.
Representation learning turns pixels into a smaller useful vector.
prior (noun): An assumed probability distribution over a variable before seeing data.
The prior over z is usually a simple Gaussian.
multimodal (adjective): Having several distinct possible outcomes rather than just one.
A conditional VAE can represent multimodal policies.
sequence VAE (noun): A VAE that models an entire sequence of latent states over time.
A sequence VAE treats a whole trajectory as the latent variable.
state space model (noun): A model that represents how a hidden state evolves over time and produces observations.
Sequence VAEs are used as state space models for planning.
structured (adjective): Organized according to a specific pattern rather than left generic.
The prior is a learned, structured transition model.
latent space (noun): The space of hidden values a model uses to represent data compactly.
The VAE learns to use the latent space efficiently.
unit-variance (adjective): Having a spread of exactly one around the mean.
The encoder's output is pushed toward a unit-variance prior.
incentivize (verb): To create a reason or reward that encourages a certain behavior.
The KL term incentivizes the encoder to use the latent space well.
plausible (adjective): Reasonable and likely to be true or valid.
Points sampled from the prior decode into plausible outputs.
downstream (adjective): Relating to a later task that depends on earlier processing.
A disentangled representation can help a downstream Q-function.
computation graph (noun): A diagram showing the order of operations used to compute a result.
The VAE training objective has a specific computation graph.
factor of variation (phrase): One independent way in which data can differ, such as position or lighting.
A VAE can separate factors of variation like position and lighting.
per-timestep (adjective): Applied separately at each individual time step.
The sequence VAE uses a per-timestep decoder.
trajectory (noun): A full sequence of states over time.
The sequence VAE treats a whole trajectory as the latent variable.
infer (verb): To work out something not directly observed, from available evidence.
The encoder infers z_t from the history of observations.
planning (noun): Choosing a sequence of actions ahead of time to reach a goal.
Sequence VAEs have been used for planning with learned state space models.
actor-critic (noun): An RL method combining a policy and a value estimator trained together.
Stochastic Latent Actor-Critic uses VAE representations for images.
pixel (noun): One tiny colored dot that makes up a digital image.
The VAE learns z instead of using raw pixels directly.
efficient (adjective): Achieving a result without wasting resources or space.
The encoder is incentivized to use the latent space efficiently.
transition (noun): A change from one state to the next.
The learned prior models the transition p(z_{t+1}|z_t,a).
architecture (noun): The overall structure and design of a neural network.
The VAE machinery is applied to several concrete architectures.

← Lecture 18, Variational Inference, Part 3 · Lecture 19, Control as Inference, Part 1 →