Lecture 13, Part 6: Exploration

What this lecture covers

This final part of Lecture 13 covers the third class of deep RL exploration methods: information gain. It addresses the practical question of what to gain information about (state density, or more usefully, the transition dynamics) and how to approximate the otherwise intractable information-gain computation, focusing on the variational inference approach used in the paper VIME.

The lecture closes the exploration topic with a recap of the three method families covered across the lecture and a reading list for further study.

Key ideas

• Information gain about dynamics: since reward is often sparse, information gain about the transition model p(s'|s,a) is a more useful target than information gain about the reward itself.
• Prediction gain: a crude approximation to information gain, computed as the change in log-density of a state before and after updating a density model on it, similar in spirit to pseudo-counts.
• Information gain as KL divergence: information gain about a variable z from an observation y equals the KL divergence between the posterior p(z|y) and the prior p(z).
• Variational approximation (VIME): since the true posterior over dynamics-model parameters is intractable, approximate it with a variational distribution q(theta|phi) (for example, independent Gaussians per parameter) and measure information gain as the KL divergence between the variational posterior before and after a new transition.
• Bayesian neural network updates: after each transition, update the variational parameters phi to phi'; the KL divergence between the two Gaussian posteriors has a closed form and serves as the exploration bonus.
• Simplifying to model error: dropping the full Bayesian treatment and just measuring how much a model's parameters or predictions change is a related, simpler family of error-based novelty bonuses.
• Recap: the lecture reiterates the three method families covered across Lecture 13 -- optimistic exploration (counts and pseudo-counts), Thompson sampling (bootstrapped Q-function ensembles), and information gain (variational approximations).

Before you watch

• Watch the earlier parts of Lecture 13, especially the pseudo-counts and bootstrapped-ensemble methods, since this part builds on and contrasts with both.
• Familiarity with KL divergence and the basic idea of variational inference is helpful, though the lecture summarizes what is needed.

Check your understanding

1. Why is information gain about the dynamics generally more useful for exploration than information gain about the reward?
2. How does prediction gain approximate information gain without explicitly computing pseudo-counts?
3. Why does information gain equal the KL divergence between the posterior and prior over the variable of interest?
4. What practical difficulty motivates using a variational approximation instead of the true posterior over dynamics-model parameters?