Lecture 20: Inverse Reinforcement Learning, Part 3

What this lecture covers

Maximum entropy inverse RL, as derived earlier in this lecture, needs known dynamics and a small discrete state-action space so it can compute exact forward and backward messages. This part addresses what breaks in realistic, high-dimensional problems and builds up the fixes step by step: sampling instead of enumerating, a "lazy" policy that is only partially optimized between reward updates, and an importance-sampling correction that keeps the resulting gradient estimate unbiased.

After watching, you can explain why naive maximum entropy IRL does not scale, and describe how the guided cost learning algorithm combines these ideas into a practical, deep-learning-compatible IRL method.

Key ideas

• Scalability problems: exact maximum entropy IRL needs the soft optimal policy solved in an inner loop and requires enumerating all state-action pairs, both infeasible with unknown dynamics or large/continuous spaces.
• Sampling-based gradient: with unknown dynamics, the reward gradient's two expectations can be estimated by sampling trajectories from the expert and from the current soft-optimal policy, rather than exact enumeration.
• The scaling problem: fully re-optimizing the policy to convergence at every reward gradient step is too expensive.
• Lazy policy optimization: only partially improving the policy at each reward update reduces cost, but produces a biased gradient estimate because the policy samples no longer come from the true soft-optimal distribution.
• Importance sampling correction: weighting each sampled trajectory by w_j, proportional to the exponential of its current reward divided by its probability under the partially optimized policy, corrects the bias; weights approach 1 as the policy improves.
• Guided cost learning: alternates between updating the reward with the importance-weighted gradient and updating the policy with a maximum-entropy policy gradient, producing both a learned reward and a usable policy.

Walkthrough

Why exact maximum entropy IRL doesn't scale (0:11)

The lecture recaps what the earlier maximum entropy algorithm requires: solving for the soft optimal policy inside the training loop and normalizing over all state-action tuples. Both become infeasible once dynamics are unknown or the state-action space is large or continuous, motivating a search for tractable approximations.

From exact enumeration to sampling (1:01)

The reward gradient is still a difference between an expectation over expert trajectories and one over the current soft-optimal policy. The first term is estimated directly from expert demonstrations. For the second, the lecture proposes learning a soft optimal policy with any max-entropy RL algorithm (such as soft Q-learning) and sampling trajectories from it, but notes this requires solving the forward RL problem to convergence at every single gradient step on the reward.

Lazy policy optimization and importance sampling (3:24)

To avoid re-solving the policy from scratch each time, the policy is only improved a little between reward updates. This introduces bias, which is corrected with importance weights derived from the ratio of the exponentiated current reward to the trajectory's probability under the partially optimized policy. As reward updates continue, the policy converges toward the target distribution and the importance weights approach one, so this correction stays valid throughout training.

Guided cost learning (5:32)

These pieces combine into guided cost learning, the first deep IRL algorithm shown to scale to high-dimensional state-action spaces. It alternates: sample trajectories from the current policy, use those samples plus expert demonstrations to update the reward via the importance-weighted gradient, then update the policy with a maximum-entropy policy gradient using the new reward. The lecture closes with an example from Finn et al., where a real robot learns to pour into a cup from demonstrations, having to visually locate the cup rather than replay a fixed motion.

Before you watch

• Review the maximum entropy IRL derivation from the earlier part of this lecture, since this part builds directly on its gradient expression.
• Recall importance-weighted policy gradients from earlier in the course, since the correction here follows the same pattern.

Check your understanding

1. Why does exact maximum entropy IRL fail to scale to unknown dynamics or large state-action spaces?
2. What bias is introduced by only partially optimizing the policy between reward updates, and how is it corrected?
3. What happens to the importance weights as the policy gets closer to the soft optimal policy for the current reward?
4. What are the two alternating updates performed in the guided cost learning algorithm?