Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 54 of 99 · 19:51

Lecture 13, Part 1: Exploration

CS 285: Lecture 13, Part 1 on YouTube

Study guide

What this lecture covers

This lecture opens a new topic, exploration, using Montezuma's Revenge and the card game Mao to illustrate why temporally extended tasks with sparse, weakly correlated rewards are so hard for RL agents: without dense feedback, an agent can get stuck exploiting a known-small reward (like repeatedly dying to respawn near a key) instead of discovering better strategies.

The lecture then frames exploration formally, from the exploration-exploitation trade-off to a spectrum of problem settings, from theoretically tractable multi-armed bandits up to the large, mostly intractable MDPs used in deep RL. It ends by defining bandits and regret precisely, setting up the algorithms covered in the next part.

Key ideas

  • Sparse and weak reward signal: tasks like Montezuma's Revenge give little guidance about whether an action was good, so trial-and-error alone struggles to find long action sequences that lead to reward.
  • Exploration vs. exploitation: exploitation repeats the best-known action; exploration tries new behaviors that might be better; the two are not mutually exclusive, since exploiting can be the best way to reach new states worth exploring from.
  • Regret against a Bayes-optimal agent: an exploration strategy's optimality is measured by how much worse it does than a hypothetical, intractable Bayesian agent that always makes the information-optimal decision.
  • Tractability spectrum: multi-armed bandits (one step, no state) are the most tractable to analyze, followed by contextual bandits (one step, with state), then small finite MDPs, then the large or continuous MDPs used in deep RL, where little can be proven theoretically.
  • Bandit as a POMDP: viewing the unknown reward parameters as a hidden state turns bandit exploration into a partially observed MDP, where the belief over those parameters evolves as you pull arms.
  • Regret: the gap between the expected reward of the best possible action, taken repeatedly, and the actual cumulative reward received while exploring.

Before you watch

  • Recall Q-learning from the deep RL value-based lectures, since Montezuma's Revenge is used as an example of where that homework implementation struggles.
  • No prior exposure to bandits or POMDPs is assumed; both are introduced from scratch.

Check your understanding

  1. Why is finishing Montezuma's Revenge only weakly correlated with the reward signal the agent receives?
  2. Why are exploration and exploitation not strictly opposite choices in a task like navigating between rooms in Montezuma's Revenge?
  3. How does casting a multi-armed bandit as a POMDP turn a one-step decision problem into a multi-step one?
  4. What does it mean for an exploration strategy to have low regret relative to a Bayes-optimal agent?

Vocabulary

exploration (noun)
The act of trying new actions or situations to discover more about the world.
Exploration is needed to find rooms far from the start.
sparse reward (phrase)
A reward signal given only rarely, with long stretches of no feedback.
Montezuma's Revenge has a sparse reward signal.
weakly correlated (phrase)
Only loosely related, without a strong connection.
Reaching the goal is only weakly correlated with early actions.
trial and error (phrase)
Learning by repeatedly trying things and observing what works.
Trial and error alone struggles with long action sequences.
exploitation (noun)
Choosing the action currently believed to give the best outcome.
Exploitation repeats the best-known action.
respawn (verb)
To reappear back in the game after dying or failing.
The agent keeps dying to respawn near a key.
regret (noun)
The gap between the best possible outcome and what was actually achieved.
Regret measures how much worse an agent did than optimal.
Bayes-optimal (adjective)
Making the mathematically best decision using all available probability information.
A Bayes-optimal agent always makes the information-optimal choice.
tractable (adjective)
Solvable or analyzable with a reasonable amount of effort.
Bandits are the most tractable exploration setting.
multi-armed bandit (phrase)
A simple decision problem with several choices and no changing state.
A multi-armed bandit has one step and no state.
contextual bandit (phrase)
A bandit problem where the decision also depends on some observed context.
A contextual bandit adds state information to the bandit setting.
finite MDP (phrase)
A decision process with a limited, countable number of states and actions.
Small finite MDPs are more tractable than large ones.
hidden state (phrase)
An unknown internal condition that affects outcomes but cannot be directly observed.
The unknown reward parameters act as a hidden state.
belief (noun)
An agent's current estimate or probability distribution about an unknown quantity.
The agent's belief over reward parameters evolves as it pulls arms.
cumulative reward (phrase)
The total reward added up over all time steps so far.
Regret compares cumulative reward against the best possible.
temporally extended (phrase)
Spanning many time steps rather than happening quickly.
Temporally extended tasks are especially hard to explore.
mutually exclusive (phrase)
Unable to both be true or happen at the same time.
Exploration and exploitation are not mutually exclusive.
motivating example (phrase)
An illustrative case used to introduce and justify a topic.
Montezuma's Revenge is used as a motivating example.
spectrum (noun)
A range of related things arranged from one extreme to another.
The lecture frames a spectrum of problem settings.
illustrate (verb)
To show or clarify an idea using a concrete example.
Mao illustrates the difficulty of sparse rewards.

Chapters

← Lecture 12, Part 4: Model-Based RL with Policies · Lecture 13, Part 2: Exploration →