Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 16 of 99 · 13:17

Lecture 5, Part 2: Intuition and the High-Variance Problem

CS 285: Lecture 5, Part 2 on YouTube

Study guide

What this lecture covers

This part builds intuition for the policy gradient formula derived earlier in the lecture and shows why it doesn't work well out of the box. It compares the policy gradient to maximum likelihood training, extends the derivation to continuous actions, notes that the derivation carries over to partially observed problems, and then demonstrates a core weakness: the policy gradient estimator has high variance.

After watching, you can explain the policy gradient as a reward-weighted version of maximum likelihood, write the log-probability gradient for a Gaussian policy, and describe why adding a constant to the rewards can change the estimated gradient for a finite sample even though it shouldn't change the optimal policy.

Key ideas

  • Weighted maximum likelihood: grad log pi is the same term used in supervised maximum likelihood training; policy gradient multiplies it by trajectory reward, so it raises the log-probability of high-reward trajectories and lowers it for low-reward ones, instead of always raising it.
  • Discrete actions: log pi is simply the log probability the network assigns to the chosen discrete action.
  • Continuous actions: representing pi_theta(a|s) as a Gaussian with a neural-network mean gives a closed-form gradient, -1/2 * Sigma^-1 * (s - a) * d(mean)/d(theta), computed by backpropagation.
  • Trial-and-error formalized: the policy gradient can be read as gradient ascent that makes "good" trajectories more likely and "bad" ones less likely, formalizing trial-and-error learning.
  • Partial observability: the derivation never used the Markov property, so the same policy gradient formula applies unchanged to observations instead of states in partially observed problems.
  • High variance: with a small number of samples, shifting all rewards by a constant (which should not change the optimal policy) can visibly change the estimated gradient direction, because it changes which samples get their probability raised versus lowered; this variance shrinks only as sample count grows.

Before you watch

  • Watch the previous part of this lecture, which derives the policy gradient formula and the REINFORCE algorithm.
  • Know the Markov property and how it differs between states and observations.

Check your understanding

  1. How does the policy gradient differ from the maximum likelihood gradient used in supervised learning?
  2. Why does adding a constant offset to all rewards leave the optimal policy unchanged in theory, but still change the estimated gradient with a small sample?
  3. Why does the policy gradient derivation extend to partially observed problems without modification?
  4. What is the closed-form gradient of the log probability for a Gaussian policy with a network-predicted mean?

Vocabulary

intuition (noun)
A natural feeling for how something works, before working through the exact math.
This lecture builds intuition for the policy gradient.
out of the box (phrase)
Working correctly without any special adjustment.
Plain policy gradient doesn't work well out of the box.
maximum likelihood (noun)
A training method that adjusts a model to make the observed data as probable as possible.
Policy gradient resembles a reward-weighted maximum likelihood.
reward-weighted (adjective)
Scaled according to how much reward was received.
The policy gradient is a reward-weighted version of the log-probability gradient.
log probability (noun)
The logarithm of how likely an outcome is under a model.
The gradient depends on the log probability of the chosen action.
Gaussian policy (noun)
A policy that chooses actions by sampling from a bell-shaped probability distribution.
Continuous actions are often modeled with a Gaussian policy.
closed-form (adjective)
Expressed as an exact formula rather than an approximation.
The Gaussian policy has a closed-form gradient.
trial-and-error (noun)
Learning by repeatedly trying actions and observing outcomes.
Policy gradient formalizes trial-and-error learning.
partial observability (noun)
A situation where the full state of the environment cannot be directly seen.
The policy gradient derivation still works under partial observability.
high variance (noun)
A large amount of randomness or spread in an estimate across different samples.
The core weakness of policy gradient is its high variance.
offset (noun)
A constant amount added to or subtracted from a set of values.
Adding a constant offset to rewards shouldn't change the optimal policy.
estimator (noun)
A rule or formula used to calculate an approximate value from data.
The policy gradient estimator has high variance with few samples.
raise (verb)
To increase the value or probability of something.
Policy gradient raises the probability of high-reward trajectories.
lower (verb)
To decrease the value or probability of something.
Low-reward trajectories have their probability lowered.
backpropagation (noun)
A method for computing gradients by passing them backward through a network.
The Gaussian policy's gradient is computed by backpropagation.
network-predicted (adjective)
Produced as the output of a neural network.
The mean is a network-predicted value in a Gaussian policy.
finite sample (noun)
A limited, countable set of collected data points.
A finite sample can give a noisy gradient estimate.
unchanged (adjective)
Staying exactly the same, with no modification.
The optimal policy stays unchanged when rewards shift by a constant.
flaw (noun)
A weakness or fault in something.
High variance is a core flaw of plain policy gradient.
mean (noun)
The central, average value that a distribution is centered around.
A neural network predicts the mean of the Gaussian policy.

Chapters

← Lecture 5, Part 1: Deriving the Policy Gradient · Lecture 5, Part 3: Reducing Variance with Causality and Baselines →