Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 7 of 99 · 8:31

Lecture 2: Imitation Learning, Part 4

CS 285: Lecture 2, Imitation Learning. Part 4 on YouTube

Study guide

What this lecture covers

This short part introduces goal-conditioned behavioral cloning, a way to squeeze more useful training signal out of demonstration data by labeling each trajectory with the state it ended up reaching, rather than only with a single fixed task.

After watching, you should be able to explain how goal relabeling works, why it can improve coverage of the state space, and how it can be used for iterative self-improvement.

Key ideas

  • The problem with single-goal data: training a policy to reach one fixed location (P1) from optimal demonstrations gives limited coverage of the state space, making compounding errors more likely.
  • Goal relabeling: instead of assuming a fixed goal, treat whatever state a demonstration ended up reaching as its implicit goal, and train a policy conditioned on both the current state and that final state.
  • More usable data: this lets a policy learn from demonstrations that were suboptimal for one goal but successful for another, since every trajectory becomes a valid example for reaching wherever it ended.
  • A theoretical caveat: goal relabeling introduces a second source of distributional shift beyond the usual training/rollout mismatch, which the lecture leaves as an open question, though the method often still works well in practice.
  • Online self-improvement: a policy can be bootstrapped by collecting data from random goal-directed rollouts, relabeling that data with the goals actually reached, retraining, and repeating — improving without a hand-designed reward function.

Before you watch

  • Watch Lecture 2 Parts 1 through 3 first; this part builds directly on behavioral cloning and the multimodal/non-Markovian issues discussed earlier.

Check your understanding

  1. How does goal relabeling turn a suboptimal demonstration into useful training data?
  2. What is the second source of distributional shift that goal-conditioned behavioral cloning introduces?
  3. How can goal-conditioned behavioral cloning be applied iteratively as a form of self-improvement?

Vocabulary

goal-conditioned (adjective)
Depending on a specific target or goal that is given as input.
Goal-conditioned behavioral cloning uses the target state as input.
relabel (verb)
To assign a new label to existing data after collecting it.
We relabel each trajectory with the goal it actually reached.
coverage (noun)
How much of the possible range of situations a dataset represents.
Goal relabeling improves coverage of the state space.
implicit (adjective)
Understood without being directly stated.
The reached state becomes the trajectory's implicit goal.
suboptimal (adjective)
Not the best possible, but still usable.
Suboptimal demonstrations can still be useful data.
caveat (noun)
A warning about a limitation of a method.
There's a theoretical caveat with goal relabeling.
bootstrap (verb)
To build something up gradually using its own early results.
The policy can bootstrap itself from its own rollouts.
self-improvement (noun)
A process where a system gets better using its own generated data.
Online self-improvement retrains the policy repeatedly.
hand-designed (adjective)
Created manually by a person rather than learned automatically.
This method works without a hand-designed reward function.
iterative (adjective)
Repeating a process multiple times to gradually improve a result.
The method uses an iterative loop of collecting and retraining.
open question (noun)
A problem that has not yet been fully solved or understood.
The theoretical issue is left as an open question.
trajectory (noun)
The sequence of states and actions a policy produces over time.
Each trajectory ends at some final state.
demonstration (noun)
A recorded example of a task being performed, used to train a policy.
Even an unsuccessful demonstration can be useful data.
state space (noun)
The full set of possible states a system could be in.
Better coverage of the state space reduces compounding errors.
fixed (adjective)
Set in advance and not changing.
Training toward one fixed goal limits what the policy learns.
usable (adjective)
Able to be used effectively for a purpose.
Goal relabeling makes more of the collected data usable.
reach (verb)
To arrive at a particular state or location.
The policy learns to reach whichever state the demonstration ended at.
mismatch (noun)
A difference between two things that should ideally match.
Distributional shift is a mismatch between training and test data.
in practice (phrase)
In real, applied use rather than only in theory.
Goal relabeling often works well in practice.
online (adjective)
Happening live, during ongoing interaction, rather than from a fixed dataset.
Online self-improvement collects new data as it goes.

Chapters

← Lecture 2: Imitation Learning, Part 3 · Lecture 2: Imitation Learning, Part 5 →