Lecture 20: Inverse Reinforcement Learning, Part 1

What this lecture covers

Every reinforcement learning method covered so far has assumed a reward function is handed to the agent. This lecture asks what to do when a good reward function is hard to write by hand but demonstrations of good behavior are available, and introduces inverse reinforcement learning (IRL) as the problem of recovering a reward function from those demonstrations. It sits at the start of a two-lecture block that connects the control-as-inference framework from the previous lecture to concrete IRL algorithms.

After watching, you can state the IRL problem formally, explain why it is inherently underspecified, and describe the classic feature-matching and maximum-margin approaches along with their main limitations.

Key ideas

• Inverse RL problem: given trajectories sampled from an (assumed) optimal policy, learn a reward function r_psi that explains them, rather than learning a policy from a known reward.
• Imitation vs. intent: humans imitating others copy the perceived intent behind an action, not the exact motions, which motivates recovering a reward rather than just cloning actions.
• Reward ambiguity: for any observed behavior there are infinitely many reward functions that would make that behavior optimal, so IRL needs extra assumptions to disambiguate.
• Linear and neural reward parametrizations: classic IRL uses a reward that is a weighted combination of features (psi^T f); modern deep IRL replaces this with a neural network mapping states and actions to a scalar reward.
• Feature matching: an early disambiguation strategy that picks a reward whose optimal policy has the same expected feature values as the expert, though multiple reward vectors can still satisfy this.
• Maximum margin principle: borrowed from SVMs, it picks the reward that makes the expert's policy better than all others by the largest possible margin, weighted by how different a policy is from the expert's.
• Shortcomings of margin-based IRL: the margin criterion is a heuristic with no explicit model of expert sub-optimality, and it leads to a messy constrained optimization that does not scale well to neural network rewards.

Walkthrough

Motivation and human behavior as optimal decision-making (1:00)

The lecture opens by framing IRL as the problem of learning a reward function instead of a policy. It connects this to a long tradition in psychology and neuroscience of explaining human motion and decision-making through models of optimal or near-optimal behavior, noting that a soft (stochastic) model of optimality fits human behavior better than a strictly deterministic one.

Practical motivations: imitation learning and RL perspectives (4:02)

Two justifications are given for learning rewards. From imitation learning, copying an expert's intent rather than their exact actions produces more natural generalization, illustrated with a psychology experiment where a child infers an experimenter's goal rather than mimicking a failed attempt literally. From an RL perspective, some tasks (games with a visible score) have obvious rewards, but others, such as autonomous driving, balance many competing factors that are far easier to demonstrate than to write down as an equation.

Defining IRL and why it is ambiguous (8:13)

Using a simple grid-world example, the lecture shows that many different reward functions, from "reward for reaching one square" to "reward negative infinity for any action outside the demonstration," can all explain the same short trajectory equally well. This motivates a formal side-by-side definition of forward RL (given a reward, learn pi*) versus inverse RL (given trajectories from pi*, learn the reward), along with linear and neural reward parametrizations.

Feature matching and the maximum margin principle (13:18)

The lecture surveys pre-deep-learning IRL methods. Feature matching picks a reward so the learned policy's expected features match the expert's, but remains ambiguous. The maximum margin principle, borrowed from SVMs, resolves this by maximizing the gap between the expert's policy value and all others, weighted by a divergence measure between policies. The lecture closes by naming the formulation's core weaknesses: the margin is a heuristic without a real model of expert sub-optimality, and the resulting constrained optimization does not extend cleanly to neural network reward functions, setting up the probabilistic approach covered next.

Before you watch

• Review the control-as-inference and soft optimality material from the previous lecture, since this lecture reuses that graphical model to define IRL probabilistically.
• Be comfortable with the basic idea of a linear reward as a weighted sum of features.

Check your understanding

1. Why is inverse reinforcement learning described as an underspecified problem?
2. How does copying an expert's intent differ from standard behavioral cloning?
3. What does the maximum margin principle add on top of simple feature matching?
4. What are the two main shortcomings of margin-based IRL methods mentioned at the end of the lecture?