Lecture 2: Imitation Learning, Part 2

What this lecture covers

This part gives the formal explanation for why behavioral cloning is not guaranteed to work, building on the informal argument from Part 1. It introduces distributional shift, defines a cost function counting mistakes, and derives a bound on the expected number of mistakes a cloned policy makes over a trajectory.

After watching, you should be able to explain distributional shift between the training and test distributions of a policy, describe the "tightrope walker" worst-case example, and state the resulting error bound and why it matters.

Key ideas

• Distributional shift: a policy is trained under the demonstrator's observation distribution p_data(o_t) but evaluated under its own distribution p_{pi_theta}(o_t), and these differ because the policy doesn't drive exactly like the demonstrator.
• A cost function for mistakes: define cost as 0 if the policy matches the expert's deterministic action and 1 otherwise; the goal is to minimize expected cost under the policy's own state distribution, not the training distribution.
• The tightrope walker example: a worst-case scenario where any mistake puts the agent in a state the expert never visited, so all subsequent time steps also become mistakes; this produces an expected error on the order of epsilon * T^2, quadratic in trajectory length T.
• General bound: even without assuming the pathological tightrope setup, the expected number of mistakes for behavioral cloning is bounded by roughly epsilon * T^2 in the worst case, derived via total variation divergence between the training and rollout state distributions.
• Why quadratic growth is bad: a linear growth in errors with trajectory length would be expected and tolerable, but quadratic growth means long-horizon tasks accumulate error much faster, explaining why naive behavioral cloning degrades badly on longer trajectories.
• Why practice is less pessimistic: the tightrope example is pathological because a single mistake is unrecoverable; real tasks and datasets that include recovery behavior (like the earlier left/right camera trick) let policies learn to correct small errors, keeping them closer to linear error growth.
• A paradox: training data that includes mistakes and their recovery can make behavioral cloning more robust than "perfect" demonstration data, because it broadens the training distribution to cover near-mistake states.

Before you watch

• Watch Lecture 2 Part 1 first; this part relies directly on its notation (states, observations, cost) and its informal compounding-error argument.
• Comfort with basic probability notation (expectations, conditional distributions) will help with the derivation.

Check your understanding

1. What is distributional shift, and why does behavioral cloning suffer from it while standard supervised learning does not?
2. In the tightrope walker example, why does a single mistake lead to roughly T additional mistakes?
3. What is the overall worst-case bound on expected mistakes for behavioral cloning, and why is quadratic growth in T a problem?
4. Why might training data containing mistakes and recoveries actually improve a cloned policy compared to "perfect" demonstrations?