Lecture 13, Part 2: Exploration

What this lecture covers

Continuing the exploration topic, this part presents three families of provably good strategies for the multi-armed bandit setting introduced previously: optimistic exploration (upper confidence bounds), posterior sampling (Thompson sampling), and information-gain-based methods. Each is a practical, tractable stand-in for solving the full POMDP formulation of the bandit exactly.

The lecture explains the intuition and rough guarantees behind each strategy and closes by noting the common thread: every method needs some form of uncertainty estimate and some assumption about the value of new information, since you cannot directly optimize toward reward when you don't yet know where it is.

Key ideas

• Optimistic exploration (UCB): pick the action maximizing the estimated mean reward plus a bonus proportional to uncertainty; a simple bonus of sqrt(2 log t / N(a)) achieves the asymptotically optimal O(log t) regret.
• Optimism in the face of uncertainty: try an arm until you are confident it is not great, then stop; unknown arms are assumed to possibly be good.
• Thompson sampling (posterior sampling): maintain a belief distribution over the bandit's unknown parameters, sample one hypothesis, act optimally under it, then update the belief from the observed outcome.
• Model-based vs. model-free: UCB is largely model-free (it just counts pulls), while Thompson sampling and information-gain methods explicitly maintain a belief over the underlying model.
• Information gain: quantifies how much observing a variable y (such as a reward) is expected to reduce the entropy of an unknown quantity z (such as model parameters), used to prefer actions that teach you the most.
• Information-directed sampling: a decision rule that trades off expected sub-optimality (delta_a) against information gain (g_a), roughly minimizing delta_a^2 / g_a, so as to avoid wasted or uninformative exploration.
• Common assumption across all three: since you can't directly chase reward you don't yet understand, each method substitutes a proxy value for new information -- optimism assumes unknowns might be good, Thompson sampling trusts its sampled model, and information gain values learning itself.

Before you watch

• Watch the first part of Lecture 13, which defines bandits, regret, and the exploration-exploitation trade-off used throughout this part.

Check your understanding

1. Why does adding an uncertainty-scaled bonus to the estimated mean reward lead to a provably good exploration strategy?
2. How does Thompson sampling differ from directly solving the bandit's POMDP formulation?
3. What is information gain measuring, and why is it defined as an expectation over the unknown observation y?
4. What assumption do optimism, Thompson sampling, and information gain each make about the value of unknown outcomes?