Lecture 6, Part 2: Discount Factors in Actor-Critic

What this lecture covers

Continuing directly from Part 1, this part assembles the pieces discussed so far into a complete batch actor-critic algorithm: sample trajectories, fit a value function, compute advantages, form the policy gradient, and take a gradient step. It then addresses a problem that arises in continuing (infinite-horizon) tasks: bootstrapped value updates can grow without bound.

You come away able to state the five-step batch actor-critic algorithm, explain why a discount factor is introduced, and distinguish two ways of adding discounting to a Monte Carlo policy gradient, including why one of them is generally preferred in practice.

Key ideas

• Batch actor-critic algorithm: generate rollouts, fit V(s) to sampled rewards, compute A(s,a) = r + V(s') - V(s) for each transition, form the policy gradient from advantages, then update the policy.
• Infinite-horizon problem: repeatedly bootstrapping value estimates in a task with no fixed end can push values toward infinity, especially when rewards are always positive.
• Discount factor (gamma): a number between 0 and 1 (commonly 0.9 to 0.999) that down-weights future rewards, keeping value estimates finite.
• Death-state interpretation: gamma can be viewed as adding a fifth "death" state to the MDP that the agent enters with probability 1 - gamma at each step, after which reward is always zero.
• Two ways to discount the Monte Carlo policy gradient: multiplying only the reward-to-go by gamma^(t'-t) (matches the critic version), versus discounting the whole trajectory from the start, which also discounts the gradient term itself by gamma^(t-1).
• Preferred choice: the first option is generally used in practice, because it avoids treating later decisions as less important, which is not the behavior wanted for continuing tasks like locomotion.
• Online actor-critic: with a discount factor, actor-critic can run fully online, updating the value function and policy after every single environment step using only the current transition (s, a, s', r).

Before you watch

• Watch Part 1 of this lecture, since it derives the advantage function and value-fitting targets used here.
• Recall the causality trick from earlier policy gradient material, since it is used to compare the two discounting options.

Check your understanding

1. Why can bootstrapped value estimates diverge in an infinite-horizon task without discounting?
2. Explain the "death state" interpretation of the discount factor.
3. What is the mathematical difference between discounting only the reward-to-go versus discounting the entire trajectory, and why does it matter?
4. Why is the fully online actor-critic algorithm able to update using only s, a, s', and r, without needing further future states?
5. What trade-off does introducing a discount factor make between bias and variance?