Lecture 14, Part 2: Learning Goal-Reaching Policies Without Rewards

What this lecture covers

This part of the lecture tackles a hard question in unsupervised reinforcement learning: how can an agent learn without any reward function by proposing its own goals and then trying to reach them? It sits within a broader discussion of exploration and unsupervised RL, following an earlier lecture on novelty-seeking exploration bonuses such as pseudo-counts.

The lecture builds up a concrete algorithm, called skew-fit, step by step: use a generative model to imagine goals, train a goal-conditioned policy to reach them, and reweight the training data so the generative model keeps proposing harder, more diverse goals. By the end, you can see why this recipe is equivalent to maximizing the mutual information between the states an agent reaches and the goals it is given.

Key ideas

• Goal-conditioned policy: a policy pi(a | s, g) that takes both the current state and a target goal state as input, so one policy can attempt many different goals.
• Latent goal space: because raw images or states are too high-dimensional to compare directly, a generative model (the lecture uses a variational autoencoder as an example) maps states to a compact latent vector z, so similar states end up close together.
• Goal proposal loop: sample a latent z from the generative model, decode it into an imagined goal image, have the policy attempt to reach it, and use the resulting data to update both the policy and the generative model.
• Skewed fitting: instead of fitting the generative model with plain maximum likelihood, weight each visited state by its density raised to a negative power, which upweights rarely seen states.
• Entropy increase: this skewed weighting provably increases the entropy of the goal distribution each iteration, pushing it toward a uniform distribution over valid states.
• Mutual information view: the overall objective works out to maximizing the mutual information between the final state and the goal, which combines high goal entropy (broad coverage) with low conditional entropy of the goal given the final state (accurate goal reaching).

Walkthrough

Setting up the problem and the generative model (0:00)

The lecture opens by restating the robot-in-the-kitchen scenario: an agent should practice unsupervised during the day and be ready to accomplish whatever goal it is given in the evening, where a goal is specified as a target state or image. Because comparing raw images directly does not scale, the lecture introduces a generative model with a latent representation z, using a variational autoencoder as a running example, and notes that other generative models would work just as well.

Sketching the basic algorithm (4:03)

The lecture sketches a first version of the algorithm: sample a latent goal z_g from the prior, decode it into an imagined goal image x_g, attempt to reach it with a goal-conditioned policy trained with something like Q-learning, and use the resulting data to update both the policy and the generative model. It then points out the flaw in this naive version: the generative model only ever sees data similar to what the agent has already done, so it keeps proposing similar goals and the agent's repertoire never expands.

Skew-fit: reweighting goals for coverage (8:05)

To fix the coverage problem, the lecture introduces the skew-fit idea: instead of standard maximum likelihood, fit the generative model with a weighted maximum likelihood objective where each observed state is weighted by its own density raised to a negative power. This is directly analogous to count-based exploration bonuses. The lecture explains, without a full proof, that this weighting guarantees the entropy of the goal distribution increases every iteration, eventually approaching a uniform distribution over valid states.

The mutual information objective (12:10)

The lecture then asks what objective this whole procedure is really optimizing. It shows that the goal distribution's entropy increases (good coverage) while the policy improving at reaching goals means the final state predicts the goal more precisely, so the conditional entropy of the goal given the final state decreases. Together these two effects mean the algorithm maximizes the mutual information between the goal and the state reached, which the lecture frames as a general and elegant way to quantify exploration performance.

Robot door-opening example (14:10)

The lecture closes with a real robot experiment where a robotic gripper is placed in front of a door with no reward signal telling it to open it. Over training the robot's self-proposed goals become progressively more ambitious, and its behavior moves from random wiggling to reliably opening the door to different angles, eventually generalizing to open the door to whatever angle appears in a given goal image.

Before you watch

• This lecture assumes familiarity with Q-learning and policy training from earlier lectures in the course.
• It references the previous lecture's discussion of novelty-seeking exploration bonuses (counts and pseudo-counts), so watching that part first will make the analogy clearer.
• Basic familiarity with the idea of a variational autoencoder is useful, though the lecture only uses it as an illustrative example.

Check your understanding

1. Why is comparing raw states directly insufficient for proposing goals in high-dimensional or continuous state spaces?
2. What problem arises if the generative model is fit with plain maximum likelihood on the agent's visited states, and how does skew-fit address it?
3. Explain why maximizing the mutual information between the goal and the final state captures both diversity and accuracy in goal reaching.
4. What did the door-opening robot experiment demonstrate about how goal proposals change over training?