Lecture 8, Part 1: Replay Buffers and the Correlation Problem

What this lecture covers

Opening Lecture 8, this part recaps the general fitted Q-iteration recipe and its online special case, Q-learning, then digs into a practical problem beyond the non-convergence issue raised previously: taking gradient steps on sequential, highly correlated transitions makes the function approximator locally overfit to whatever part of the trajectory it just saw, then "forget" as the trajectory moves on.

You come away understanding two solutions: parallel data collection across multiple workers (as used in actor-critic), and the more common approach of a replay buffer, which stores past transitions and samples i.i.d. batches from them to decorrelate updates and provide broader data coverage.

Key ideas

• Online Q-learning recap: take one action, compute one target value y = r + gamma * max_a' Q(s', a'), and take one gradient step toward it, repeating this every environment step.
• Correlation problem: consecutive states in a trajectory are highly correlated, so training on them one at a time causes the function approximator to locally overfit to recent transitions and lose accuracy on earlier parts of the trajectory.
• Parallel workers: running multiple simulators and collecting a batch of transitions from all of them at once reduces correlation, similar to the parallel actor-critic approach from an earlier lecture.
• Replay buffer: a stored collection of past transitions from potentially many different policies; sampling a batch i.i.d. from the buffer decorrelates training samples and lowers gradient variance.
• Buffer needs refreshing: because an early, poor policy may not visit useful parts of the state space, the buffer must periodically be replenished by deploying the latest policy (often with epsilon-greedy exploration) to collect new transitions.
• Full deep Q-learning recipe: collect transitions into a buffer, sample a batch, compute target values and take a gradient step summed over the batch, and repeat some number of times before collecting more data.

Before you watch

• Watch the earlier parts of this lecture sequence on fitted Q-iteration and Q-learning, since this part builds directly on that algorithm and target value computation.
• Recall the discussion of synchronous and asynchronous parallel actor-critic, since a similar parallelism idea is reused here.

Check your understanding

1. Why does training on sequential transitions one at a time cause the function approximator to overfit locally?
2. How does using multiple parallel workers help address the correlation problem?
3. Why does sampling i.i.d. batches from a replay buffer decorrelate training samples?
4. Why is it still necessary to periodically collect new data with the latest policy rather than relying only on the initial buffer contents?