Lecture 2: Imitation Learning, Part 5

What this lecture covers

This closing part of the imitation learning lecture presents DAgger (Dataset Aggregation), an algorithm that fixes distributional shift by relabeling states the policy actually visits, then explains why imitation learning by itself is insufficient, motivating the shift to reward-based reinforcement learning.

After watching, you should be able to describe the DAgger loop, explain why it provably bounds error linearly rather than quadratically in trajectory length, and state why RL avoids imitation learning's dependence on costly human-labeled data.

Key ideas

• DAgger's core idea: instead of changing the policy to fit the data, change the data to match the policy's own state distribution by running the current policy, collecting the states it visits, and asking a human to label each with the correct action.
• The DAgger loop: train on the initial demonstrations, run the policy, label the new observations, aggregate this data with the original set, and retrain — repeating until the training distribution converges toward the policy's rollout distribution.
• Theoretical guarantee: unlike naive behavioral cloning's quadratic-in-T error bound, DAgger provably achieves a bound linear in T, at the cost of requiring ongoing human labeling.
• A practical limitation: labeling states offline, out of the flow of actually performing the task, can be unnatural for humans (e.g. labeling a driving image without the real-time context of reacting while driving).
• Why imitation learning alone isn't enough: it requires plentiful human-provided data, which is expensive and limits scale; humans are also poor at specifying some kinds of low-level actions (e.g. rotor commands for aerobatics); and autonomous data collection allows machines to self-improve beyond human performance.
• Toward reward-based objectives: the lecture reframes the same imitation-learning cost function in terms of a reward function, setting up the transition to defining more general reward functions for reinforcement learning.

Before you watch

• Watch Lecture 2 Parts 1 through 4 first; DAgger is presented as a direct fix to the compounding-error problem developed there.

Check your understanding

1. What does DAgger change about the data collection process compared to standard behavioral cloning?
2. Why does DAgger achieve a linear rather than quadratic error bound?
3. What practical difficulty can arise when asking humans to label states offline in step three of DAgger?
4. Why might autonomous data collection be preferable to human demonstrations for some robotic tasks?