Lecture 13, Part 3: Exploration

What this lecture covers

This part moves from bandit theory to practical exploration algorithms for deep RL, recapping the three families introduced earlier (optimism, Thompson sampling, information gain) and focusing on optimistic, count-based methods. It shows how to add a UCB-style exploration bonus to the reward using state visitation counts, then confronts the core problem: exact counts are nearly useless once state spaces are large or continuous, since the same exact state is almost never seen twice.

The solution presented is pseudo-counts: fitting a density model to observed states and deriving a quantity that behaves like a count from how that density changes as new states are added, following the "unifying count-based exploration" approach. The lecture also shows empirical results on Atari, including a large improvement on Montezuma's Revenge.

Key ideas

• Exploration bonus: augment the reward with r_plus = r + bonus(N(s)), where the bonus decreases as a state's visitation count grows, extending the bandit UCB idea to MDPs.
• The trouble with counts: in large or continuous state spaces, combinatorial variation (or truly continuous states) means the exact same state is rarely, if ever, revisited, making literal counting ineffective.
• Density models as pseudo-counts: fit a density model p_theta(s) to visited states; a state similar to previously seen states gets high density even if never seen exactly before.
• Deriving pseudo-counts: by requiring the density model to satisfy the same count-probability relationships as real counts, you can solve two equations for the pseudo-count n_hat(s) from the density before and after updating on a new state.
• Choosing a density model: unlike typical generative modeling, pseudo-counts only need a density score, not good samples or even normalized probabilities, which changes which models are suitable (the original paper uses a simple pixel-conditional CTS model).
• Empirical impact: on Montezuma's Revenge, adding a 1/sqrt(N(s))-style bonus lets the agent explore far more rooms of the pyramid than plain Q-learning, though the effect varies by game.

Before you watch

• Watch the previous parts of Lecture 13 on bandit exploration strategies, since the UCB bonus and regret framing are carried over directly.

Check your understanding

1. Why does an exploration bonus based on exact state counts fail in large or continuous MDPs?
2. How does a density model's change after observing a new state get converted into a pseudo-count?
3. Why can a pseudo-count density model be low quality at generating samples yet still work well for exploration bonuses?
4. What did the Montezuma's Revenge results suggest about the practical value of pseudo-count exploration bonuses?