Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 15 of 99 · 14:01

Lecture 5, Part 1: Deriving the Policy Gradient

CS 285: Lecture 5, Part 1 on YouTube

Study guide

What this lecture covers

This part derives the course's first concrete reinforcement learning algorithm, policy gradient, by directly differentiating the RL objective with respect to the policy parameters. It picks up from the trajectory distribution and objective defined earlier in the course and asks: how can you estimate both the objective's value and its gradient using only samples, without knowing the transition probabilities or initial state distribution?

After watching, you can evaluate the RL objective by averaging rewards over sampled rollouts, derive the policy gradient using the log-derivative ("log-grad") trick, and state the three steps of the REINFORCE algorithm.

Key ideas

  • Evaluating the objective: J(theta) can be estimated by running the policy to collect N sampled trajectories, summing rewards along each, and averaging — an unbiased estimate that improves as N grows.
  • The log-derivative trick: the identity grad p(tau) = p(tau) * grad log p(tau) lets you rewrite grad J(theta) as an expectation, so it can be estimated from samples the same way as the objective itself.
  • Only the policy term survives: expanding log p(tau) with the chain rule shows that its derivative with respect to theta depends only on sum_t grad log pi_theta(a_t | s_t), since the initial state distribution and transition probabilities don't depend on theta.
  • Policy gradient formula: grad J(theta) is the expectation, over sampled trajectories, of (sum_t grad log pi_theta(a_t | s_t)) * (sum of rewards along the trajectory).
  • REINFORCE algorithm: (1) sample trajectories by running the policy, (2) estimate the policy gradient with the formula above, (3) take a gradient ascent step theta <- theta + alpha * grad J(theta).
  • Anatomy mapping: sampling trajectories is the orange box, summing rewards is the green box, and the gradient ascent step is the blue box, matching the generic RL algorithm anatomy from earlier lectures.

Before you watch

  • Review the RL objective as an expectation over the trajectory distribution and the chain-rule factorization of that distribution from the previous lecture.
  • Be comfortable with basic calculus, especially the derivative of a logarithm.

Check your understanding

  1. Why can't you compute grad p(tau) directly, and what identity lets you get around this?
  2. Which terms in log p(tau) vanish when you take the gradient with respect to theta, and why?
  3. Write out the three steps of the REINFORCE algorithm.
  4. How is the objective J(theta) estimated from a finite set of sampled trajectories?

Vocabulary

differentiate (verb)
To calculate the derivative of a function.
The lecture differentiates the RL objective with respect to theta.
sample-based (adjective)
Estimated using collected data samples rather than an exact formula.
A sample-based gradient estimate avoids needing the true dynamics.
unbiased estimate (noun)
An estimate whose average value equals the true quantity being estimated.
Averaging sampled rewards gives an unbiased estimate of the objective.
log-derivative trick (noun)
An identity that rewrites the derivative of a probability as the probability times the derivative of its log.
The log-derivative trick makes the gradient estimable from samples.
identity (noun)
A mathematical equation that is always true.
The log-derivative identity holds for any probability distribution.
vanish (verb)
To become zero and disappear from an equation.
The transition terms vanish when differentiated with respect to theta.
REINFORCE (noun)
The basic policy gradient algorithm that samples trajectories and updates the policy toward higher reward.
REINFORCE is the first concrete RL algorithm derived in the course.
gradient ascent (noun)
An optimization method that updates parameters to increase a value, like reward.
REINFORCE ends with a gradient ascent step.
learning rate (noun)
A number controlling how large each parameter update step is.
Alpha is the learning rate in the gradient ascent update.
orange box (noun)
The step in the RL anatomy where samples are collected.
Sampling trajectories is the orange box.
green box (noun)
The step in the RL anatomy where an estimate is computed.
Summing rewards is the green box in REINFORCE.
blue box (noun)
The step in the RL anatomy where the policy is improved.
The gradient ascent step is the blue box.
derive (verb)
To work out a formula step by step from known rules.
The lecture derives the policy gradient from the RL objective.
objective (noun)
The value a training process tries to maximize or minimize.
The RL objective is the expected sum of rewards.
expectation (noun)
The average outcome you would expect over many repeated trials.
The gradient is rewritten as an expectation over trajectories.
estimate (verb)
To calculate an approximate value using available data.
We estimate the objective by averaging sampled rewards.
improves as N grows (phrase)
Becomes more accurate as more samples are collected.
The estimate improves as N grows larger.
transition probabilities (noun)
The chances of moving from one state to another given an action.
The gradient doesn't require knowing the transition probabilities.
initial state distribution (noun)
The probability of starting in each possible state.
The initial state distribution doesn't depend on theta.
trajectories (noun)
Recorded sequences of states and actions produced by a policy.
N sampled trajectories are used to estimate the objective.

Chapters

← Lecture 4, Part 6: Examples of Deep RL Algorithms · Lecture 5, Part 2: Intuition and the High-Variance Problem →