Lecture 5, Part 3: Reducing Variance with Causality and Baselines

What this lecture covers

This part addresses the high-variance problem identified earlier in the lecture by introducing two practical fixes that make policy gradients usable. It answers: how can you lower the variance of the policy gradient estimator without introducing bias?

After watching, you can explain the causality trick that replaces full trajectory reward with "reward to go," derive why subtracting a baseline from the reward leaves the gradient unbiased, and describe what the variance-minimizing optimal baseline looks like.

Key ideas

• Causality: an action at time t cannot affect rewards at earlier time steps t' < t, since this is always true for any forward-flowing process (unlike the Markov property, which only sometimes holds).
• Reward to go: exploiting causality, the reward sum multiplying grad log pi at time t can be restricted to rewards from t onward, denoted Q-hat(i, t); this term is a single-sample estimate of the same quantity as the Q-function from the previous lecture.
• Lower variance from fewer terms: dropping past rewards from the sum reduces the total quantity being multiplied, which lowers the estimator's variance while keeping it unbiased.
• Baselines: subtracting a constant b (such as average reward) from the reward before multiplying by grad log pi keeps the gradient unbiased in expectation, because the extra term's expectation is provably zero — this holds for any choice of b.
• Why baselines help: without centering, an all-positive reward signal raises the probability of every trajectory, including bad ones; subtracting a baseline makes above-average trajectories more likely and below-average ones less likely, matching the intended trial-and-error behavior.
• Optimal baseline: minimizing the variance with respect to b gives b = E[g^2 * r] / E[g^2], a gradient-magnitude-weighted average reward, different for each parameter; in practice, the simpler average-reward baseline is used instead.

Before you watch

• Watch the earlier parts of this lecture deriving the policy gradient and demonstrating its high-variance problem.
• Know the identity p(tau) * grad log p(tau) = grad p(tau) used in the policy gradient derivation.

Check your understanding

1. How does causality justify dropping past rewards from the policy gradient sum, and why does this reduce variance?
2. Why does subtracting a baseline b from the reward not bias the policy gradient in expectation?
3. What form does the variance-minimizing optimal baseline take, and why is it rarely used in practice?
4. What is "reward to go," and how does it relate to the Q-function from the previous lecture?