Lecture 5, Part 1: Deriving the Policy Gradient

What this lecture covers

This part derives the course's first concrete reinforcement learning algorithm, policy gradient, by directly differentiating the RL objective with respect to the policy parameters. It picks up from the trajectory distribution and objective defined earlier in the course and asks: how can you estimate both the objective's value and its gradient using only samples, without knowing the transition probabilities or initial state distribution?

After watching, you can evaluate the RL objective by averaging rewards over sampled rollouts, derive the policy gradient using the log-derivative ("log-grad") trick, and state the three steps of the REINFORCE algorithm.

Key ideas

• Evaluating the objective: J(theta) can be estimated by running the policy to collect N sampled trajectories, summing rewards along each, and averaging — an unbiased estimate that improves as N grows.
• The log-derivative trick: the identity grad p(tau) = p(tau) * grad log p(tau) lets you rewrite grad J(theta) as an expectation, so it can be estimated from samples the same way as the objective itself.
• Only the policy term survives: expanding log p(tau) with the chain rule shows that its derivative with respect to theta depends only on sum_t grad log pi_theta(a_t | s_t), since the initial state distribution and transition probabilities don't depend on theta.
• Policy gradient formula: grad J(theta) is the expectation, over sampled trajectories, of (sum_t grad log pi_theta(a_t | s_t)) * (sum of rewards along the trajectory).
• REINFORCE algorithm: (1) sample trajectories by running the policy, (2) estimate the policy gradient with the formula above, (3) take a gradient ascent step theta <- theta + alpha * grad J(theta).
• Anatomy mapping: sampling trajectories is the orange box, summing rewards is the green box, and the gradient ascent step is the blue box, matching the generic RL algorithm anatomy from earlier lectures.

Before you watch

• Review the RL objective as an expectation over the trajectory distribution and the chain-rule factorization of that distribution from the previous lecture.
• Be comfortable with basic calculus, especially the derivative of a logarithm.

Check your understanding

1. Why can't you compute grad p(tau) directly, and what identity lets you get around this?
2. Which terms in log p(tau) vanish when you take the gradient with respect to theta, and why?
3. Write out the three steps of the REINFORCE algorithm.
4. How is the objective J(theta) estimated from a finite set of sampled trajectories?