Lecture 14, Part 4: Learning Diverse Skills

What this lecture covers

This final part of the lecture moves beyond covering individual states or goals to learning genuinely diverse skills or behaviors. It addresses a gap left by the earlier goal-reaching and state-coverage methods: not every useful behavior can be expressed as reaching a single goal state, since some tasks require visiting a whole region of the state space in a particular way while avoiding other regions.

The lecture introduces a skill-conditioned policy pi(a | s, z), where z indexes a skill, and shows how a diversity-promoting reward built from a discriminator can train many distinct skills purely from unsupervised interaction. It ends by tying this method, like the two before it, back to a mutual information objective, closing out the lecture's broader theme.

Key ideas

• Skill-conditioned policy: a policy pi(a | s, z) where z is a task or skill index (for example, one of n discrete skills), generalizing the idea of separate policies for separate behaviors into one conditional policy.
• Skills versus goals: goal-reaching policies can only aim for a single target state, so they cannot express behaviors like "reach the green ball while avoiding the red circle"; diverse skills need to visit distinct regions of state space, not just distinct states.
• Diversity-promoting reward: reward each skill z for visiting states that are unlikely under other skills, so that observing the states an agent visits reveals which skill it was running.
• Discriminator as reward: a classifier p(z | s) is trained to predict which skill produced a given state, and the reward for the policy is log p(z | s), so the policy is pushed toward being easy to identify from its behavior.
• Co-adaptation loop: the discriminator and skills bootstrap off each other, starting from skills that differ only by chance and progressively separating further as the classifier's decision boundary sharpens.
• Mutual information connection: the objective decomposes as H(z) - H(z | s); a uniform prior over skills maximizes the first term, and training the discriminator and policy well minimizes the second, so the whole procedure maximizes the mutual information between skill and state.

Walkthrough

From goals to skills (0:00)

The lecture introduces the skill-conditioned policy pi(a | s, z) and explains that discrete skills (such as a fixed set of behaviors like washing dishes) are a special case of this general form. It gives a 2D navigation example where different skill indices correspond to moving in different directions, then argues that reaching diverse goals is not the same as performing diverse tasks, since goal-conditioned policies cannot express behaviors that require avoiding certain regions rather than just reaching a target.

Building a diversity-promoting reward (2:02)

The lecture proposes rewarding each skill for visiting states that are unlikely under other skills, framed as a classification problem: train a discriminator that predicts which skill z produced a given state, and reward the policy with the log-probability the discriminator assigns to the correct skill. It walks through a visualization with two skills (green and blue) that start out doing similar things by chance, where the discriminator's decision boundary and the policies' divergence reinforce each other over training, eventually producing clearly separated behaviors such as running forward, running backward, and performing a flip on a benchmark cheetah task.

Connecting to mutual information (5:04)

The lecture closes by showing that this method also reduces to maximizing mutual information, this time between the skill z and the state s: choosing a uniform prior over skills maximizes the entropy term H(z), while training the discriminator and policy to make skills predictable from state minimizes H(z | s). It then draws together all three methods covered across this lecture (goal proposal, state marginal matching, and diverse skills), noting that each is, in its own way, an instance of maximizing mutual information between an outcome and a notion of goal or task, and that this is provably the right approach when the test-time task is chosen adversarially.

Before you watch

• Watch Lecture 14, Parts 2 and 3 first, since this segment explicitly builds on and ties together the goal-reaching and state marginal matching methods discussed there.
• Familiarity with the concept of mutual information and its decomposition into entropy terms is helpful, as it appears in this and the two preceding parts.

Check your understanding

1. Why can't a goal-conditioned policy express a behavior like "reach a target while avoiding an obstacle," and how does a skill-conditioned policy address this?
2. How does the discriminator's reward encourage different skills to visit different regions of the state space?
3. Walk through why maximizing log p(z | s) corresponds to minimizing the conditional entropy H(z | s).
4. How do the goal-proposal, state-marginal-matching, and diverse-skills methods from this lecture all relate to maximizing mutual information?