Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 56 of 99 · 14:31

Lecture 13, Part 3: Exploration

CS 285: Lecture 13, Part 3 on YouTube

Study guide

What this lecture covers

This part moves from bandit theory to practical exploration algorithms for deep RL, recapping the three families introduced earlier (optimism, Thompson sampling, information gain) and focusing on optimistic, count-based methods. It shows how to add a UCB-style exploration bonus to the reward using state visitation counts, then confronts the core problem: exact counts are nearly useless once state spaces are large or continuous, since the same exact state is almost never seen twice.

The solution presented is pseudo-counts: fitting a density model to observed states and deriving a quantity that behaves like a count from how that density changes as new states are added, following the "unifying count-based exploration" approach. The lecture also shows empirical results on Atari, including a large improvement on Montezuma's Revenge.

Key ideas

  • Exploration bonus: augment the reward with r_plus = r + bonus(N(s)), where the bonus decreases as a state's visitation count grows, extending the bandit UCB idea to MDPs.
  • The trouble with counts: in large or continuous state spaces, combinatorial variation (or truly continuous states) means the exact same state is rarely, if ever, revisited, making literal counting ineffective.
  • Density models as pseudo-counts: fit a density model p_theta(s) to visited states; a state similar to previously seen states gets high density even if never seen exactly before.
  • Deriving pseudo-counts: by requiring the density model to satisfy the same count-probability relationships as real counts, you can solve two equations for the pseudo-count n_hat(s) from the density before and after updating on a new state.
  • Choosing a density model: unlike typical generative modeling, pseudo-counts only need a density score, not good samples or even normalized probabilities, which changes which models are suitable (the original paper uses a simple pixel-conditional CTS model).
  • Empirical impact: on Montezuma's Revenge, adding a 1/sqrt(N(s))-style bonus lets the agent explore far more rooms of the pyramid than plain Q-learning, though the effect varies by game.

Before you watch

  • Watch the previous parts of Lecture 13 on bandit exploration strategies, since the UCB bonus and regret framing are carried over directly.

Check your understanding

  1. Why does an exploration bonus based on exact state counts fail in large or continuous MDPs?
  2. How does a density model's change after observing a new state get converted into a pseudo-count?
  3. Why can a pseudo-count density model be low quality at generating samples yet still work well for exploration bonuses?
  4. What did the Montezuma's Revenge results suggest about the practical value of pseudo-count exploration bonuses?

Vocabulary

exploration bonus (phrase)
An extra reward added to encourage visiting less-seen states.
The exploration bonus decreases as a state gets visited more.
visitation count (phrase)
The number of times a state has been visited so far.
The bonus depends on a state's visitation count.
combinatorial (adjective)
Involving a huge number of possible combinations.
Combinatorial variation makes exact states rarely repeat.
density model (phrase)
A model that estimates how likely or common a given input is.
A density model gives high density to similar states.
pseudo-count (noun)
An estimated count for a state derived from a density model, standing in for a real count.
Pseudo-counts approximate real visitation counts.
generative modeling (phrase)
Building a model that can produce new, realistic samples similar to training data.
This differs from typical generative modeling goals.
density score (phrase)
A number representing how likely a data point is under a model.
Pseudo-counts only need a density score, not good samples.
normalized (adjective)
Adjusted so probabilities correctly add up to one.
The model doesn't even need normalized probabilities.
pixel-conditional (adjective)
Predicting each pixel based on the pixels before it.
The original paper uses a pixel-conditional CTS model.
empirical impact (phrase)
A real, measured effect seen in experiments.
The empirical impact varies a lot by game.
pyramid (noun)
A structure of rooms or levels arranged like a pyramid shape, as in the game's map.
The agent explores far more rooms of the pyramid.
trouble (noun)
A difficulty or problem that arises.
The lecture explains the trouble with literal counts.
confront (verb)
To directly face and deal with a difficult problem.
This part confronts the core problem with exact counts.
literal (adjective)
Exact, taken precisely as it is, without approximation.
Literal counting fails in large state spaces.
derive (a formula) (verb)
To work out a formula step by step from basic principles.
The lecture derives pseudo-counts from two equations.
unifying (adjective)
Bringing several separate ideas together under one framework.
This follows the unifying count-based exploration approach.
suitable (adjective)
Appropriate or well matched for a purpose.
This changes which models are suitable for pseudo-counts.
vary (by game) (verb)
To differ from one case to another.
The effect varies by which Atari game is used.
extend (a bonus) (verb)
To make an idea apply to a broader or different setting.
This extends the bandit UCB idea to full MDPs.
practical algorithm (phrase)
A method that actually works well when implemented in real systems.
The lecture moves toward practical algorithms for deep RL.

Chapters

← Lecture 13, Part 2: Exploration · Lecture 13, Part 4: Exploration →