Lecture 5, Part 6: The Natural Policy Gradient

What this lecture covers

This closing part of the lecture diagnoses a numerical problem that hurts policy gradients, especially with continuous actions, and derives a fix called the natural (or covariant) policy gradient. It uses a small 1D example — a Gaussian policy with a mean parameter k and a variance parameter sigma — to show why plain gradient ascent struggles, then reframes gradient ascent as a constrained optimization problem to motivate the fix.

After watching, you can explain why some policy parameters need much smaller learning rates than others, describe gradient ascent as maximizing a linearized objective within a parameter-space trust region, and explain how replacing that region with a KL-divergence constraint on the policy distribution produces the natural gradient.

Key ideas

• Poor conditioning example: in a small Gaussian-policy problem, the gradient with respect to sigma grows much larger than the gradient with respect to k as sigma shrinks, so the raw gradient direction doesn't point toward the optimum — the same issue as optimizing a quadratic with a badly scaled Hessian.
• Gradient ascent as constrained optimization: a first-order gradient step can be viewed as maximizing a linear (Taylor) approximation of the objective subject to a small step-size constraint ||theta' - theta||^2 <= epsilon in parameter space.
• Wrong space for the constraint: constraining step size in raw parameter space is awkward, because some parameters change the policy distribution a lot per unit step and others change it very little.
• KL-divergence constraint: replacing the parameter-space constraint with a constraint on KL(pi_theta' || pi_theta) (a parameterization-independent divergence between distributions) makes step sizes consistent in policy space rather than parameter space.
• Fisher information matrix: a second-order Taylor expansion of the KL divergence gives a quadratic form using the Fisher information matrix F = E[grad log pi * grad log pi^T], estimated from samples.
• Natural gradient update: solving the constrained problem gives the update theta <- theta + alpha * F^-1 * grad J(theta); multiplying by F^-1 rescales the gradient so it points toward the optimum, converging faster and requiring less step-size tuning. Natural policy gradient and TRPO are named algorithms that use this idea, with TRPO deriving alpha from a chosen epsilon via conjugate gradient.

Before you watch

• Watch the earlier parts of this lecture, especially the policy gradient derivation and implementation.
• Basic familiarity with Taylor expansions, constrained optimization (Lagrange multipliers), and KL divergence is helpful.

Check your understanding

1. Why does the raw policy gradient point in a poor direction when one parameter (like sigma) has a much larger gradient magnitude than another?
2. How does viewing gradient ascent as a constrained optimization problem motivate replacing the parameter-space constraint with a KL-divergence constraint?
3. What role does the Fisher information matrix play in the natural gradient update?
4. What is the practical benefit of using F^-1 * grad J(theta) instead of the raw gradient?