Lecture 2: Imitation Learning, Part 3

What this lecture covers

This part surveys practical techniques that make behavioral cloning work well in practice, beyond the theory covered earlier. It explains why data with intentional mistakes and corrections helps, why treating actions as Markovian or unimodal can fail, and how to represent richer action distributions using mixtures of Gaussians, latent variable models, diffusion models, and autoregressive discretization.

After watching, you should be able to explain why imperfect demonstration data can outperform perfect data, describe non-Markovian and multimodal behavior as failure modes for naive behavioral cloning, and name at least three ways to represent multimodal action distributions.

Key ideas

• Intentional mistakes and corrections: deliberately including mistakes and their corrections in demonstration data teaches a policy to recover from errors, because corrections dilute mistakes less than they help the model generalize to near-error states.
• Data augmentation as a related trick: synthetic corrective examples, like the drone-through-forest paper's three head-mounted cameras labeled left/right/straight, achieve a similar effect without needing real recovery demonstrations.
• Non-Markovian behavior: human demonstrators often act based on more than the current observation (memory of what they saw or reacted to earlier), so a policy that only sees the current frame may fail to reproduce their behavior; feeding in a history of observations through a sequence model (LSTM, Transformer) can help, though it risks causal confusion, where spurious correlations (e.g. a brake light visible in the cabin) are mistaken for causes of the correct action.
• Multimodal behavior: when multiple different actions are equally valid (e.g. going left or right around a tree), a single Gaussian output collapses these into an invalid average; the lecture presents three fixes — mixtures of Gaussians, latent variable models (e.g. conditional VAEs), and diffusion models.
• Diffusion policies: modeled on image diffusion, a network learns to predict and remove noise added to expert actions; at test time it starts from random noise and iteratively denoises it into a clean action, conditioned on the observation.
• Autoregressive discretization: rather than discretizing a high-dimensional action space directly (which is exponential in the number of bins), each action dimension is discretized and predicted one at a time through a sequence model, keeping the number of bins needed linear in dimensionality.
• Case studies: diffusion policies (Chi et al., 2023) for robot manipulation, latent-variable Transformer policies with a learned "style" variable for tasks like shoe-buckling, and RT-1's autoregressive per-dimension discretization for language-conditioned robot control.

Walkthrough

What makes behavioral cloning easy and what makes it hard (0:42)

The lecture recaps why perfect demonstration data is fragile and explains how intentionally collected mistakes and corrections make a policy more robust to compounding errors.

Trail following as classification and imitation with a cheap robot (2:51)

Two case studies illustrate the point: a drone navigation system trained with three head-mounted cameras labeled by direction, and a low-cost teleoperated robot arm whose imperfect demonstrations teach it to recover from disturbances.

Why might we fail to fit the expert? (6:52)

The lecture identifies non-Markovian behavior (expert actions depending on history, not just the current observation) as one reason a policy fails to reproduce expert behavior, and shows how reading in a history of frames through a sequence model can help, while cautioning about causal confusion from spurious correlations like a visible brake indicator.

How can we use the whole history? / Expressive continuous distributions (9:00)

After covering history-based policies, the lecture turns to multimodal behavior, where several different actions are equally valid, and introduces mixtures of Gaussians, latent variable models (conditional VAEs), and diffusion models as ways to represent multimodal continuous action distributions.

Autoregressive discretization (25:14)

The lecture explains discretizing one action dimension at a time through a sequence model, so the number of bins grows linearly rather than exponentially with dimensionality, and shows how the chain rule of probability justifies this as a valid way to sample from the full action distribution.

Imitation with diffusion models, latent variables and Transformers (28:15)

Three case studies close the lecture: a diffusion-based policy for robot manipulation, a latent-variable Transformer policy for tasks like buckling a shoe, and RT-1, which uses autoregressive discretization with language conditioning to perform diverse, instruction-following robot tasks.

Before you watch

• Watch Lecture 2 Parts 1 and 2 first; this part assumes familiarity with behavioral cloning and the compounding-error problem.
• Basic familiarity with sequence models (LSTMs, Transformers) and generative models (VAEs, diffusion models) helps with the second half of the lecture, though the lecture introduces them at a survey level.

Check your understanding

1. Why can demonstration data containing mistakes and corrections sometimes produce a better policy than perfectly optimal demonstrations?
2. What is causal confusion, and how might reading in a history of observations make it worse?
3. Why does outputting a single Gaussian distribution fail for tasks like flying around a tree?
4. How does autoregressive discretization avoid the exponential blow-up of naive discretization in high-dimensional action spaces?
5. What role does the diffusion process play in a diffusion policy, both during training and at test time?