Lecture 13, Part 5: Exploration

What this lecture covers

This short part adapts Thompson sampling (posterior sampling) to deep RL. Since a bandit's reward model becomes a Q-function in an MDP, the lecture describes sampling a Q-function from an ensemble, acting on it for a full episode, then updating the ensemble on the collected data, since Q-learning is off-policy and can train on data from any exploration strategy.

Key ideas

• Q-function as the sampled model: in deep RL, posterior sampling means maintaining a distribution over Q-functions and acting greedily on one sampled function per episode, rather than sampling reward parameters as in bandits.
• Bootstrap ensembles: approximate a posterior by resampling the dataset with replacement to train several independent Q-networks, then sampling one at random to act on.
• Shared-body, multi-head networks: a cheaper approximation trains one network with a shared trunk and multiple output heads, giving correlated but still somewhat varied behavior across heads.
• Why it beats random exploration: acting on one consistent sampled Q-function for a whole episode produces coherent behavior (such as reliably surfacing for air in Seaquest), unlike epsilon-greedy, which rarely chains together the many correct random actions needed for such a strategy.
• Trade-offs: bootstrapped exploration needs no reward modification and no exploration-exploitation hyperparameter, but tends to underperform well-tuned count-based or pseudo-count bonuses, and does poorly on very hard exploration problems like Montezuma's Revenge.

Before you watch

• Watch the earlier parts of Lecture 13 on Thompson sampling in bandits and on count-based and pseudo-count exploration, which this method is compared against.

Check your understanding

1. Why can Q-learning train on data collected under many different sampled Q-functions without modification?
2. How does acting on one sampled Q-function per episode help with games like Seaquest, where a long consistent action sequence is required?
3. What is the practical trade-off between bootstrapped exploration and bonus-based methods like pseudo-counts?