Lecture 9, Part 1: Why Does Policy Gradient Work?

What this lecture covers

This opening part of Lecture 9 asks a foundational question: given that policy gradient is just gradient ascent, why should we actually expect it to improve the policy? It reframes the REINFORCE-style policy gradient recipe as an instance of policy iteration, then works through a derivation showing that improving the RL objective from an old policy to a new one is mathematically equivalent to maximizing the expected advantage of the old policy under the new policy's trajectory distribution.

This builds directly on the basic policy gradient and actor-critic material from earlier in the course, and specifically revisits the policy iteration algorithm from the value-based methods lecture. It sets up the rest of Lecture 9, which will address the remaining gap in the proof: how close the new and old policies' state distributions need to be for this argument to hold.

Key ideas

• Policy gradient recap: sample trajectories, estimate the reward-to-go (Q-hat) via Monte Carlo or a learned value function, and take a gradient ascent step on grad log pi * Q-hat.
• Generalized view: policy gradient can be seen as alternating between estimating an advantage A-hat for the current policy and using it to produce an improved policy, the same generate-fit-improve pattern used elsewhere in the course.
• Policy gradient as softened policy iteration: policy iteration jumps directly to the argmax action for each state; policy gradient instead nudges probabilities toward high-advantage actions without jumping all the way, which is reasonable when the advantage estimate itself is imperfect.
• The central claim: the improvement in the RL objective, J(theta') - J(theta), equals the expected value, under the new policy's trajectory distribution, of the sum of discounted advantages of the old policy.
• Why the claim matters: if maximizing that right-hand expression with respect to theta' is equivalent to maximizing J(theta'), then using the old policy's advantage to search for a new policy is a mathematically justified way to do policy improvement.
• Proof technique: rewrite the value function difference using a telescoping sum over discounted value-function terms, which regroups into a sum of one-step advantages of the old policy, expressed as an expectation under the new policy's trajectories.
• Remaining gap: turning this into a usable policy gradient requires importance-sampling actions from the old policy, but the outer expectation over states is still with respect to the new (not-yet-known) policy theta', which is only justified when theta and theta' are close, a claim proven later in the lecture.

Walkthrough

Recap and motivation (1:15)

The lecture reviews the basic policy gradient recipe, generating trajectory samples, estimating reward-to-go, and taking a gradient step, and frames the day's goal as understanding why this procedure reliably improves the policy, using tools from policy iteration to build intuition and give the analysis relevance to actor-critic methods too.

Policy gradient as a softened policy iteration (2:40)

The lecture restates policy gradient conceptually as alternating advantage estimation and policy improvement, the same structure as policy iteration from the value-based methods lecture. The key difference is that policy iteration assigns probability 1 to the argmax action, while policy gradient's grad log pi * A-hat update only shifts probability mass gradually toward high-advantage actions. This gentler update is presented as sensible protection against an imperfect advantage estimate.

Deriving the connection between the RL objective and advantage (7:09)

The lecture sets out to prove that J(theta') - J(theta) equals the expectation, under the new policy's trajectories, of the discounted sum of the old policy's advantages. It explains why this claim matters: if maximizing the right-hand side with respect to theta' is equivalent to maximizing J(theta'), then policy iteration's practice of improving a policy using the old policy's advantage is justified as genuinely optimizing the RL objective, not just a heuristic.

Working through the proof (12:12)

Starting from the RL objective written as an expectation of the initial-state value function, the derivation substitutes in the new policy's trajectory distribution (valid because both distributions share the same initial-state marginal), then uses a telescoping-sum trick, writing the value function as a difference between two infinite discounted sums that cancel except for the leading term. Rearranging and pairing terms produces a sum of one-step differences of the form gamma * V(s_{t+1}) - V(s_t), which combine with the reward term from J(theta')'s definition to form exactly the advantage function of the old policy, evaluated under states and actions from the new policy's trajectory distribution. This completes the proof.

From the proof to a policy gradient expression (18:21)

Having established the identity, the lecture rewrites it as a sum over time steps of an expectation under the new policy's state-action marginal of the old policy's advantage. Applying the importance-sampling trick from the earlier policy gradient lecture lets the inner expectation over actions be rewritten using the old policy plus an importance weight, closely resembling the standard policy gradient expression. What remains different, and unresolved at this point, is that the outer expectation over states is still taken under the new policy's state distribution rather than the old one actually available for sampling.

The open question of distribution mismatch (20:02)

The lecture ends by naming the gap explicitly: using states sampled from the old policy in place of the new policy's state distribution is only valid when the two policies are close. Whether, and precisely when, this substitution is valid is the question the next part of the lecture addresses.

Before you watch

• Review the basic policy gradient (REINFORCE) and actor-critic material covered earlier in the course, since this lecture builds directly on the reward-to-go and advantage estimation ideas.
• Recall the policy iteration algorithm from the value-based methods lecture, which this lecture reinterprets policy gradient in terms of.

Check your understanding

1. In what sense is policy gradient a "softened" version of policy iteration?
2. State the central identity relating J(theta') - J(theta) to the advantage of the old policy, and explain in your own words why proving it matters.
3. What is the key algebraic trick used to turn a value-function difference into a sum of advantages?
4. What is the remaining obstacle to turning this identity into a practical policy gradient update, and under what condition does the lecture suggest it can be resolved?