Lecture 14, Part 3: State Marginal Matching and Intrinsic Motivation

What this lecture covers

Continuing the discussion of unsupervised exploration, this part asks what happens when you use intrinsic motivation bonuses like negative log-density purely to maximize state coverage, with no task reward at all. It generalizes the idea of maximizing state entropy to the more general problem of matching a target distribution over states, called state marginal matching.

The lecture shows that naively applying novelty-seeking rewards produces a "tail chasing" policy that never settles into good coverage, then derives a fix based on returning a mixture of policies collected across training, justified through a connection to self-play in game theory. It closes by discussing why matching a uniform target distribution is actually the right choice when nothing is known about the eventual goal.

Key ideas

• Intrinsic motivation: reward bonuses such as -log p(s) (closely related to counts and pseudo-counts) that reward visiting states with low estimated density, used even in the total absence of any task reward.
• Tail-chasing problem: repeatedly updating the policy to maximize a novelty bonus and refitting the density estimator to match it produces a density estimator with good coverage, but the final policy itself only visits an arbitrary, narrow set of states.
• State marginal matching: the more general problem of training a policy so its state marginal distribution p_pi(s) matches a target distribution p*(s) by minimizing their KL divergence; maximizing state entropy is the special case where p* is uniform.
• Why RL alone doesn't solve it: the reward log p* - log p_pi is constructed so the RL objective equals the KL divergence, but the RL algorithm does not account for the fact that the reward itself depends on the policy through p_pi, so directly optimizing it does not converge to state marginal matching.
• Mixture policy fix: instead of returning the latest policy, fit the density estimator to the union of all states seen so far, and return a mixture (randomly chosen iterate) of all the policies trained during learning.
• Self-play and Nash equilibrium: the fix works because the state distribution matching p* is the Nash equilibrium of a two-player game between the density estimator and the policy, and only the historical average of iterates (not the final iterate) converges to that equilibrium.
• Eisenbach's theorem: if a test-time adversary can choose the worst possible goal, the best training distribution to prepare for is the uniform distribution over goals or states, which justifies maximizing coverage as a training strategy even without knowing the eventual task.

Walkthrough

From novelty bonuses to state marginal matching (0:00)

The lecture opens by connecting back to Monday's exploration bonuses, describing intrinsic motivation as rewarding states with low estimated density. It walks through what happens if you alternate updating the policy to maximize this bonus and refitting the density estimator: the density estimator ends up with good coverage, but the policy itself just chases whichever region was least visited, producing arbitrary and unstable behavior rather than a policy that reliably covers the state space.

Defining the state marginal matching objective (3:02)

The lecture formalizes the goal as minimizing the KL divergence between the policy's state marginal and a target distribution, and shows that a reward built from log p* - log p_pi makes the RL objective mathematically equal to that KL divergence. It then explains the subtlety: because standard RL treats the reward as fixed rather than policy-dependent, this naive approach still produces the same tail-chasing failure.

Fixing it with mixtures and self-play (6:05)

The lecture introduces two changes: fit the density estimator to all states seen across every iteration (not just the latest), and return a mixture policy that randomly selects among all past iterates rather than only the final one. It explains why this works using game theory: the target distribution is the Nash equilibrium of a game between the policy and the density estimator, and self-play theory shows that the average of iterates, not any single iterate, converges to that equilibrium. An ant-maze experiment illustrates that this mixture approach achieves much more even coverage than a standard RL algorithm.

Why uniform coverage is a good goal (10:07)

The lecture closes by questioning the premise: is covering as many states as possible actually a good exploration objective? It presents what it calls (semi-jokingly) Eisenbach's theorem, showing that if you assume an adversary will choose the worst possible goal at test time, the optimal training distribution to prepare for that is the uniform distribution. This gives a principled justification for maximizing entropy or coverage during unsupervised training when the eventual task is unknown.

Before you watch

• Watch the previous part of this lecture (Lecture 14, Part 2) first, since this segment builds directly on the goal-proposal and generative-model ideas introduced there.
• Review the count-based and pseudo-count exploration bonuses from the earlier exploration lecture, since intrinsic motivation here is presented as a variant of those ideas.
• Basic familiarity with KL divergence and Nash equilibria is helpful for following the state marginal matching derivation.

Check your understanding

1. Why does maximizing a novelty bonus like -log p(s) fail to produce a policy with good state coverage, even though the density estimator itself ends up well-calibrated?
2. What two changes convert the naive intrinsic motivation algorithm into one that actually performs state marginal matching?
3. How does the two-player game and self-play argument justify returning a mixture of policies rather than the final policy?
4. What does Eisenbach's theorem say about the best training distribution when the test-time goal is chosen adversarially, and why does that justify uniform coverage?