Lecture 13, Part 1: Exploration

What this lecture covers

This lecture opens a new topic, exploration, using Montezuma's Revenge and the card game Mao to illustrate why temporally extended tasks with sparse, weakly correlated rewards are so hard for RL agents: without dense feedback, an agent can get stuck exploiting a known-small reward (like repeatedly dying to respawn near a key) instead of discovering better strategies.

The lecture then frames exploration formally, from the exploration-exploitation trade-off to a spectrum of problem settings, from theoretically tractable multi-armed bandits up to the large, mostly intractable MDPs used in deep RL. It ends by defining bandits and regret precisely, setting up the algorithms covered in the next part.

Key ideas

• Sparse and weak reward signal: tasks like Montezuma's Revenge give little guidance about whether an action was good, so trial-and-error alone struggles to find long action sequences that lead to reward.
• Exploration vs. exploitation: exploitation repeats the best-known action; exploration tries new behaviors that might be better; the two are not mutually exclusive, since exploiting can be the best way to reach new states worth exploring from.
• Regret against a Bayes-optimal agent: an exploration strategy's optimality is measured by how much worse it does than a hypothetical, intractable Bayesian agent that always makes the information-optimal decision.
• Tractability spectrum: multi-armed bandits (one step, no state) are the most tractable to analyze, followed by contextual bandits (one step, with state), then small finite MDPs, then the large or continuous MDPs used in deep RL, where little can be proven theoretically.
• Bandit as a POMDP: viewing the unknown reward parameters as a hidden state turns bandit exploration into a partially observed MDP, where the belief over those parameters evolves as you pull arms.
• Regret: the gap between the expected reward of the best possible action, taken repeatedly, and the actual cumulative reward received while exploring.

Before you watch

• Recall Q-learning from the deep RL value-based lectures, since Montezuma's Revenge is used as an example of where that homework implementation struggles.
• No prior exposure to bandits or POMDPs is assumed; both are introduced from scratch.

Check your understanding

1. Why is finishing Montezuma's Revenge only weakly correlated with the reward signal the agent receives?
2. Why are exploration and exploitation not strictly opposite choices in a task like navigating between rooms in Montezuma's Revenge?
3. How does casting a multi-armed bandit as a POMDP turn a one-step decision problem into a multi-step one?
4. What does it mean for an exploration strategy to have low regret relative to a Bayes-optimal agent?