Lecture 20: Inverse Reinforcement Learning, Part 4

What this lecture covers

This final part of the inverse RL lecture reframes the guided cost learning algorithm from the previous part as a game between a reward function and a policy, and shows this is formally equivalent to a generative adversarial network (GAN). It works out what discriminator structure recovers a genuine reward function versus what structure only recovers a policy, closing out the course's coverage of inverse reinforcement learning.

After watching, you can explain the correspondence between IRL and GANs, describe the special discriminator form used in adversarial IRL methods, and distinguish reward-recovering IRL from adversarial imitation learning that only recovers a policy.

Key ideas

• IRL as a game: the reward function tries to score human demonstrations high and policy samples low, while the policy tries to make its samples score as well as the demonstrations, mirroring a GAN's generator/discriminator dynamic.
• GAN basics: a generator produces samples to resemble a true data distribution, while a discriminator is trained to distinguish real data from generated samples.
• Optimal discriminator ratio: at convergence, the Bayes-optimal GAN discriminator represents the density ratio p_star / (p_star + p_theta), which reaches 0.5 exactly when the generator matches the data distribution.
• Adversarial IRL discriminator: by substituting the optimal policy's unnormalized density (p(tau) * exp(reward) / Z) into the discriminator formula, the resulting discriminator recovers an actual reward function while training exactly like a GAN.
• Partition function as a parameter: Z can be optimized jointly with the reward parameters using the same objective, removing the need for the importance-sampling correction used in guided cost learning.
• Generative adversarial imitation learning (GAIL): using an ordinary binary classifier as the discriminator instead of the special reward-based form recovers a working policy but not a reusable reward function.
• Transfer benefit of reward recovery: because IRL decouples the goal from the dynamics, a recovered reward can be re-optimized in modified conditions (for example, a robot missing two legs) to produce meaningful new behavior, which pure imitation cannot do.

Walkthrough

IRL viewed as an adversarial game (0:01)

The lecture opens by pointing out that the guided cost learning procedure already resembles a game: a reward function scores demonstrations high and policy samples low, while the policy adapts to fool that reward function. This observation is made formal by connecting inverse RL to generative adversarial networks, after a quick refresher on how GANs train a generator and discriminator against each other.

Deriving the adversarial IRL discriminator (4:03)

Substituting the soft-optimal policy's unnormalized density in place of the true data distribution in the optimal discriminator formula produces a discriminator with a specific form: the exponentiated reward divided by itself plus the policy's trajectory probability. Training this discriminator with the standard GAN objective trains the reward parameters directly, and optimizing the partition function Z alongside the reward removes the need for importance weights used in guided cost learning.

From reward recovery to pure imitation (9:06)

Adversarial IRL methods that use this special discriminator can recover rewards that generalize, demonstrated with an ant robot whose recovered reward still produces sensible behavior after two of its legs are disabled. The lecture then asks what happens if a regular binary classifier is used as the discriminator instead: this gives generative adversarial imitation learning (GAIL), which recovers a working policy but no longer yields a meaningful reward function, since the discriminator converges to a constant 0.5 with no informative structure left inside it.

Comparing methods and further reading (12:07)

The lecture closes by summarizing that guided cost learning and GAIL are structurally the same procedure, differing only in the discriminator's parametrization, one recovers a reward, the other only a policy, and points to suggested readings covering the classic feature-matching and maximum entropy methods alongside the modern deep IRL and adversarial imitation learning papers discussed across the lecture.

Before you watch

• Watch the earlier parts of this lecture on maximum entropy IRL and guided cost learning, since this part directly reframes guided cost learning as a GAN.
• Basic familiarity with how generative adversarial networks train a generator against a discriminator will help the derivation land.

Check your understanding

1. What does the optimal discriminator in a GAN converge to, and why isn't it simply 1 for all real samples?
2. How does the discriminator used in adversarial IRL differ from a standard GAN discriminator, and what does that difference buy you?
3. Why can a reward recovered through IRL generalize to a modified robot body while a cloned policy cannot?
4. What is the key structural difference between guided cost learning and generative adversarial imitation learning?