Lecture 9, Part 3: Constraining Policy Gradient with KL Divergence

What this lecture covers

Having shown that policy gradient's tractable objective is only guaranteed to improve the true RL objective when the new and old policies stay close in total variation divergence, this short part asks how to enforce that closeness in an actual algorithm. It replaces the total variation bound with the more convenient KL divergence, and shows how to enforce a KL constraint using a Lagrangian and dual gradient descent.

This directly continues from the state-distribution bound proven in Part 2 of Lecture 9, and previews the family of constrained and proximal policy gradient methods used later in the course.

Key ideas

• Why KL instead of total variation: total variation divergence is not differentiable everywhere, while KL divergence is differentiable when the two distributions share support and has tractable closed forms for many distribution families, and it upper-bounds total variation divergence, so a KL bound is a valid substitute.
• Constrained objective: maximize the importance-weighted advantage objective subject to KL(pi_theta' || pi_theta) <= epsilon, which still improves the true RL objective for small enough epsilon.
• Lagrangian formulation: form the Lagrangian by subtracting epsilon from the KL term, scale it by a multiplier lambda, and alternate maximizing over the policy parameters and taking a gradient step on lambda.
• Dual gradient descent: raising lambda when the constraint is violated and lowering it otherwise asymptotically finds a multiplier that enforces the constraint, a general technique the course revisits later.
• Heuristic alternative: instead of solving the dual problem exactly, lambda can be chosen manually and the KL term used as a fixed regularizer, penalizing deviation from the old policy, with the inner maximization run incompletely for a few gradient steps.
• Connection to real algorithms: this general recipe underlies methods such as guided policy search and PPO.

Before you watch

• Watch Part 2 of Lecture 9, which derives the total-variation bound that this section converts into a KL-divergence constraint.
• A basic familiarity with Lagrangian duality is helpful, though the lecture reviews the mechanics of dual gradient descent.

Check your understanding

1. Why is KL divergence more convenient than total variation divergence for enforcing a policy closeness constraint?
2. How does dual gradient descent adjust the Lagrange multiplier based on whether the constraint is violated?
3. What is the difference between the "principled" Lagrangian approach and the heuristic fixed-penalty approach to enforcing the KL constraint?
4. Which later algorithms does the lecture connect this constrained policy gradient recipe to?