Lecture 8, Part 4: Overestimation, Double Q-Learning, N-Step Returns

What this lecture covers

This part of Lecture 8 turns to practical questions about Q-learning: are the Q-values a trained network produces actually accurate predictions of future reward, and if not, why not? It then covers two techniques used to make Q-learning work better in practice: double Q-learning, which corrects a systematic overestimation bias, and multi-step (n-step) returns, which trade bias for variance in the learning target.

The lecture builds directly on the target-network version of Q-learning covered earlier in Lecture 8, and connects the multi-step return discussion back to the actor-critic lecture's treatment of the same bias-variance trade-off. After watching, you can explain why Q-value predictions systematically overestimate returns, describe how double Q-learning mitigates this, and construct an n-step Q-learning target.

Key ideas

• Relative accuracy: as training progresses, average predicted Q-values rise alongside average per-episode return, and on games like Breakout and Pong the value function's ups and downs track meaningful in-game events.
• Absolute inaccuracy: despite this reasonable relative behavior, predicted Q-values are systematically much larger than the actual discounted returns measured on the same policy.
• Overestimation from max: taking a max over noisy estimates has an expected value greater than or equal to the max of the true expected values, because the max preferentially selects whichever action's noise happened to be positive.
• Double Q-learning: decorrelating the network that selects the best action from the network that evaluates its value removes most of this bias; in practice this is done cheaply by using phi to select the action and the existing target network phi' to evaluate it.
• N-step returns: summing n real rewards before bootstrapping with the target network reduces reliance on a possibly inaccurate Q-function early in training, at the cost of higher variance.
• Off-policy validity: n-step returns with n > 1 are only strictly correct for on-policy data, since intermediate actions in the sum must match the current policy; n = 1 is always valid off-policy.
• Practical fixes: options for using n-step returns with off-policy data include ignoring the bias, dynamically cutting the trace to the largest n where sampled actions match the current greedy policy, or importance sampling.

Walkthrough

Are the Q-values accurate? (0:17)

The lecture asks whether a trained Q-function's predictions match reality. Learning curves on Atari games show predicted Q-values rising together with average per-episode reward, and in Breakout and Pong the value function's peaks and dips line up sensibly with in-game moments, such as anticipating a ball breaking through to the ceiling in Breakout or a return becoming urgent in Pong. This suggests the relative structure of Q-values is meaningful.

Overestimation in Q-learning (8:22)

Comparing predicted Q-values against the actual discounted sum of rewards obtained on the same policy reveals a consistent gap: predictions are systematically higher than reality. The lecture traces this to the max operator used to compute target values. Using a simple two-random-variable argument, it shows that E[max(x1, x2)] >= max(E[x1], E[x2]), because the max tends to pick whichever variable's noise pushed it upward. Since a trained Q-function can be modeled as the true value plus unbiased noise, taking a max over actions systematically selects positive noise, producing an overestimated target.

Double Q-learning in practice (14:35)

The fix is to decorrelate the noise in action selection from the noise in value evaluation. Double Q-learning uses two networks, phi_a and phi_b: one selects the action via argmax, the other evaluates it. Since Q-learning already maintains a current network phi and a target network phi', this can be implemented essentially for free by selecting the action with phi and evaluating its value with phi', rather than using phi' for both. This isn't a perfect decorrelation, since phi' is periodically copied from phi, but it removes a large fraction of the overestimation in practice.

Multi-step returns (16:17)

The standard Q-learning target uses one real reward plus a bootstrapped estimate from the next state. Early in training, when the Q-function is close to random, almost all useful learning signal comes from that single reward, and the bootstrapped term is mostly noise. As in actor-critic methods, this can be addressed by summing n discounted rewards before bootstrapping, which lowers bias (since the bootstrapped term is now scaled by gamma^n) at the cost of higher variance from using more sampled rewards.

Q-learning with n-step returns (20:02)

Because intermediate actions in an n-step sum must come from the current policy to be an unbiased estimate, n-step returns with n > 1 are technically incorrect with off-policy data, unlike n = 1. The lecture lists practical workarounds: ignore the bias (often works well in practice), dynamically choose the largest n for which the sampled actions match what the current greedy policy would have chosen, or apply importance sampling as described in the "safe and efficient off-policy reinforcement learning" paper. It leaves as an open question how a different, off-policy-friendly object could be learned in place of a plain n-step Q-function.

Before you watch

• Review the target-network version of Q-learning from earlier in Lecture 8, since double Q-learning modifies exactly that mechanism.
• Recall the bias-variance discussion of multi-step returns from the actor-critic lecture, which this section extends to Q-learning.

Check your understanding

1. Why can a Q-function's relative predictions look sensible while its absolute values are systematically wrong?
2. Explain, using the two-random-variable argument, why taking a max over noisy estimates causes overestimation.
3. How does double Q-learning use the existing phi and phi' networks to reduce overestimation without adding a third network?
4. Why are n-step returns with n > 1 only valid for on-policy data, and what are two ways to work around this?