Lecture 8, Part 2: Target Networks for Q-Learning

What this lecture covers

This part of Lecture 8 addresses a second core instability in Q-learning, after the replay buffer already fixed the problem of correlated samples. The lecture asks: why does Q-learning behave differently from ordinary regression, and how can training be stabilized when the regression targets themselves keep changing as the network updates?

By the end you can describe how a replay buffer combined with a periodically-updated target network produces the classic "deep Q-learning" (DQN) algorithm, and you understand the alternative of Polyak-averaged target updates. This builds directly on the replay buffer discussion earlier in Lecture 8 and sets up the DQN implementation used in homework 3.

Key ideas

• Moving target problem: in Q-learning the regression target depends on the same parameters being updated, so unlike supervised regression the target shifts on every gradient step, which destabilizes learning.
• Target network: a separate parameter vector phi' is used to compute target values while the online parameters phi are updated, so targets stay fixed for a stretch of training.
• General recipe: collect data with the current policy, sample a batch from the replay buffer, take k gradient steps on phi, and every n steps copy phi into phi'.
• Classic DQN: the special case where k = 1, matching the update rate of data collection and gradient steps while phi' refreshes only every n steps (often around 10,000).
• Lag inconsistency: because phi' is only replaced periodically, the target network's staleness varies from one step old to n steps old depending on where you are in the cycle.
• Polyak averaging alternative: instead of a hard copy, update phi' every step as a weighted average tau * phi' + (1 - tau) * phi with tau close to 1, giving a uniformly lagged target at every step.

Walkthrough

What's wrong with plain Q-learning (0:13)

The lecture recaps that a replay buffer solves the correlated-samples problem, but Q-learning still is not true gradient descent because its regression target moves as the parameters change. Running the regression step to full convergence on a moving target is not necessarily useful, which is why practice uses only a small number of gradient steps.

Q-learning as regression, and target networks (1:05)

The lecture connects fitted Q-iteration's regression step to this instability and introduces a hybrid: collect data continuously, but compute targets using a separate parameter vector phi' that lags behind the online parameters phi. Multiple gradient updates can be taken against the same fixed target before phi' is refreshed.

Q-learning with target networks (2:12)

The general procedure is laid out step by step: collect data, sample a batch, update phi for k steps using targets from phi', and every n steps set phi' = phi. Typical values are k between 1 and 4 and n around 10,000, chosen so target values stay stable long enough for the regression to behave like ordinary supervised learning.

The classic DQN algorithm (5:20)

Setting k = 1 in the general recipe recovers the standard deep Q-network algorithm: take a step and store it in the buffer, sample a mini-batch, compute targets with phi', update phi by regression, and every n steps synchronize phi' with phi. The lecture notes this is the version students implement for homework.

An alternative target update (8:02)

The lecture points out the oddity that periodic hard copies make the target's staleness inconsistent across steps, then introduces Polyak averaging as a smoother alternative: interpolate phi' toward phi a little every step using a large tau, which keeps the lag roughly constant and has some theoretical grounding as long as phi' stays close to phi.

Before you watch

• Know how the replay buffer removes sample correlation in Q-learning, covered earlier in Lecture 8.
• Be comfortable with fitted Q-iteration and its regression step, since target networks are framed as a middle ground between online Q-learning and full fitted Q-iteration.

Check your understanding

1. Why is Q-learning not equivalent to standard gradient descent on a fixed objective?
2. What roles do phi and phi' play in the target-network version of Q-learning?
3. How do the choices of k and n trade off data collection speed against target stability?
4. How does Polyak averaging differ from the periodic hard-copy update of the target network?