Lecture 6, Part 4: Eligibility Traces and GAE

What this lecture covers

This part returns to the bias-variance trade-off at the heart of actor-critic methods. It compares the low-variance, biased actor-critic advantage estimator against the unbiased, higher-variance Monte Carlo estimator, then shows how a state-dependent baseline (the value function) keeps the policy gradient unbiased while still reducing variance.

By the end you should be able to explain n-step returns as a way to interpolate between the two extremes, and understand how generalized advantage estimation (GAE) averages over all possible n-step returns using a single parameter.

Key ideas

• State-dependent baseline: subtracting any function of the state alone, such as the value function, keeps the policy gradient unbiased while lowering variance, unlike the biased actor-critic critic term.
• Action-dependent baseline (control variates): subtracting a Q function lowers variance further but no longer integrates to zero, requiring a correction term; this correction can sometimes be computed with low or zero variance, as in the Q-Prop method mentioned in the lecture.
• n-step returns: sum actual rewards for n steps, then bootstrap with the value function for the remainder, trading bias for variance depending on how large n is.
• Bias-variance intuition: rewards and value estimates near the present have lower variance and are more reliable, so cutting off the Monte Carlo sum earlier and relying on the value function later tends to work better than either extreme.
• Generalized advantage estimation (GAE): a weighted average of all n-step returns, using an exponential weighting controlled by a parameter lambda, analogous to a discount factor.
• GAE recursive form: GAE can be written as a discounted, lambda-weighted sum of single-step advantage terms (delta), giving an efficient way to compute the full weighted average.
• Discount and lambda as variance control: both gamma and lambda reduce variance by down-weighting distant, less certain rewards, at the cost of some bias.

Before you watch

• Watch Part 3 of this lecture on implementing actor-critic, since the advantage estimator discussed there is the baseline being improved here.
• Review the discount factor material from Part 2, since GAE's lambda plays an analogous role.

Check your understanding

1. Why does subtracting a state-dependent baseline keep the policy gradient unbiased, while subtracting an action-dependent baseline does not?
2. What does the n-step return estimator trade off as n increases?
3. Why is variance typically higher for rewards further in the future than for rewards close to the present?
4. How does GAE combine all possible n-step returns into a single estimator?
5. In what sense does the lambda parameter in GAE play a role similar to the discount factor gamma?