Lecture 2: Imitation Learning, Part 4

What this lecture covers

This short part introduces goal-conditioned behavioral cloning, a way to squeeze more useful training signal out of demonstration data by labeling each trajectory with the state it ended up reaching, rather than only with a single fixed task.

After watching, you should be able to explain how goal relabeling works, why it can improve coverage of the state space, and how it can be used for iterative self-improvement.

Key ideas

• The problem with single-goal data: training a policy to reach one fixed location (P1) from optimal demonstrations gives limited coverage of the state space, making compounding errors more likely.
• Goal relabeling: instead of assuming a fixed goal, treat whatever state a demonstration ended up reaching as its implicit goal, and train a policy conditioned on both the current state and that final state.
• More usable data: this lets a policy learn from demonstrations that were suboptimal for one goal but successful for another, since every trajectory becomes a valid example for reaching wherever it ended.
• A theoretical caveat: goal relabeling introduces a second source of distributional shift beyond the usual training/rollout mismatch, which the lecture leaves as an open question, though the method often still works well in practice.
• Online self-improvement: a policy can be bootstrapped by collecting data from random goal-directed rollouts, relabeling that data with the goals actually reached, retraining, and repeating — improving without a hand-designed reward function.

Before you watch

• Watch Lecture 2 Parts 1 through 3 first; this part builds directly on behavioral cloning and the multimodal/non-Markovian issues discussed earlier.

Check your understanding

1. How does goal relabeling turn a suboptimal demonstration into useful training data?
2. What is the second source of distributional shift that goal-conditioned behavioral cloning introduces?
3. How can goal-conditioned behavioral cloning be applied iteratively as a form of self-improvement?