Lecture 20: Inverse Reinforcement Learning, Part 2

What this lecture covers

Continuing from the introduction to inverse reinforcement learning, this part derives a concrete algorithm from the probabilistic model of optimality. It treats reward learning as maximum likelihood estimation over expert trajectories, and works through why the resulting gradient requires a hard-to-compute normalizing constant, the partition function.

After watching, you can derive the maximum entropy IRL gradient, explain how it is estimated using forward and backward messages, and describe why this formulation resolves the ambiguity that classic feature-matching IRL could not.

Key ideas

• Reward as an optimality variable: p(o_t | s_t, psi) is set to the exponential of a parameterized reward r_psi, and IRL becomes maximum likelihood estimation of psi from expert trajectories.
• Partition function (log Z): normalizing the exponentiated reward over all possible trajectories is what makes IRL hard, since it prevents the trivial solution of assigning high reward to everything.
• Gradient as a contrast: the likelihood gradient equals the expected reward gradient under the expert's trajectories minus the expected reward gradient under the current soft-optimal policy, pushing reward up on expert behavior and down elsewhere.
• State-action marginal (mu_t): computed by multiplying forward and backward messages from the previous lecture's inference algorithms, it lets the second expectation be computed without enumerating full trajectories.
• Maximum entropy IRL algorithm: iteratively compute backward and forward messages, form mu, evaluate the gradient as a difference of expectations, and take a gradient ascent step on the reward parameters until convergence.
• Disambiguation via sub-optimality: unlike feature matching, this method uses how deterministic or random the expert's behavior is to distinguish between reward functions that would otherwise explain the same trajectories equally well.
• Practical limits: the algorithm needs small, discrete state-action spaces and known transition probabilities, since it depends on tractable forward and backward message computation.

Before you watch

• Review the forward and backward message computations for soft optimal control from the previous (Monday's) lecture, since this lecture reuses them directly.
• Recall the feature-matching and maximum-margin IRL methods from Part 1, since the maximum entropy formulation is contrasted with them.

Check your understanding

1. Why does the partition function make maximum likelihood reward learning intractable in general?
2. How is the likelihood gradient expressed as a difference of two expectations, and what does each term represent?
3. Why is this approach called "maximum entropy" inverse RL, and how does it connect to feature-matching methods?
4. What assumptions about the state and action space does this algorithm rely on, and why do they limit its practical use?