Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 32 of 99 · 8:35

Lecture 8, Part 3: A Unified View of Q-Learning

CS 285: Lecture 8, Part 3 on YouTube

Study guide

What this lecture covers

This short segment steps back from the specific Q-learning variants covered so far and asks how fitted Q-iteration, online Q-learning, and DQN with replay buffers and target networks relate to each other. It reframes them as one general algorithm expressed as parallel processes running at different speeds.

After watching, you can describe Q-learning as three (or four, including buffer eviction) processes acting on a shared replay buffer, and explain each prior algorithm as a particular choice of how fast each process runs.

Key ideas

  • Shared replay buffer: the buffer of transitions is the central object that every version of Q-learning reads from and writes to.
  • Process 1 (data collection): continuously runs the current policy (for example epsilon-greedy on phi) in the environment and appends new transitions to the buffer.
  • Eviction process: keeps the buffer bounded, typically as a ring buffer that discards the oldest transition when a new one arrives.
  • Process 2 (target update): periodically copies phi into the target parameters phi', either by a hard copy or Polyak averaging, and normally runs much more slowly than the other processes.
  • Process 3 (learning): samples a batch, computes targets using phi', and updates phi by gradient descent.
  • Algorithms as rate choices: online Q-learning is the case where the buffer holds only one transition and all processes run in lockstep; DQN runs data collection and learning at the same rate with a slow, large-buffer target update; fitted Q-iteration nests learning inside target updates inside data collection.
  • Non-stationarity control: running the processes at different rates is what keeps each process's view of the problem close to stationary, which is why the separation helps convergence.

Before you watch

  • Watch the earlier parts of Lecture 8 on replay buffers and target networks, since this section reframes those exact mechanisms.

Check your understanding

  1. What are the three (or four) processes in the unified Q-learning view, and what does each one do?
  2. How does online Q-learning correspond to a specific choice of buffer size and process rates?
  3. Why does running process 2 much more slowly than process 3 help stabilize learning?
  4. How does fitted Q-iteration fit into this framework as nested loops?

Vocabulary

unify (verb)
To bring separate things together under one shared idea.
This part tries to unify several Q-learning variants.
reframe (verb)
To describe something in a new, often clearer, way.
The lecture reframes the algorithms as parallel processes.
process (noun)
A sequence of actions that runs continuously to achieve a goal.
Data collection is one process running alongside training.
replay buffer (noun)
A stored set of past experiences sampled for training.
The shared replay buffer connects all the processes.
epsilon-greedy (phrase)
A strategy that mostly picks the best action but occasionally picks a random one.
Data collection runs epsilon-greedy on the current network.
append (verb)
To add something onto the end of a list or collection.
New transitions are appended to the buffer.
eviction (noun)
The removal of old items to make space for new ones.
Eviction keeps the buffer from growing without limit.
ring buffer (phrase)
A fixed-size storage that overwrites its oldest entry when full.
The buffer works like a ring buffer, discarding the oldest transition.
hard copy (phrase)
Replacing one set of values directly with another all at once.
The target update can use a hard copy of the parameters.
rate (noun)
How fast something happens compared to something else.
Each process runs at a different rate.
lockstep (adverb)
Moving together at exactly the same pace.
In online Q-learning, all processes run in lockstep.
nest (verb)
To place one thing inside another, layer within layer.
Fitted Q-iteration nests learning inside data collection.
non-stationarity (noun)
A situation where the underlying data or rules keep changing over time.
Running processes slowly helps control non-stationarity.
stationary (adjective)
Staying the same over time, not changing.
A slow-changing target keeps the learning problem close to stationary.
gradient descent (noun)
An optimization method that repeatedly adjusts parameters to reduce error.
Process 3 updates phi by gradient descent.
sample a batch (phrase)
To randomly pick a group of examples from a larger set.
The learning process samples a batch from the buffer.
step back (phrasal verb)
To pause and look at the bigger picture instead of the details.
This part steps back to compare all the Q-learning variants.
shared object (phrase)
A single resource that multiple processes read from and write to.
The buffer is the shared object every process touches.
prior algorithm (phrase)
A method described earlier that is now being reinterpreted.
Each prior algorithm is a particular choice of process rates.
specific choice (phrase)
One particular option chosen out of several possibilities.
DQN makes a specific choice about how fast each process runs.

Chapters

← Lecture 8, Part 2: Target Networks for Q-Learning · Lecture 8, Part 4: Overestimation, Double Q-Learning, N-Step Returns →