Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 13 of 99 · 9:14

Lecture 4, Part 5: Comparing RL Algorithms

CS 285: Lecture 4, Part 5 on YouTube

Study guide

What this lecture covers

This part steps back from specific algorithms to ask why the field has so many of them instead of one best method. It answers that question by walking through the trade-offs that push you toward one algorithm family or another: how many environment samples an algorithm needs, how stable and easy to tune it is, whether it is guaranteed to converge, and what assumptions it makes about the environment.

After watching, you can explain the difference between on-policy and off-policy algorithms, place different algorithm families on a sample-efficiency spectrum, and describe which algorithm types have convergence guarantees and which do not.

Key ideas

  • Sample efficiency: how many environment samples an algorithm needs before it learns a good policy; this is the main axis separating algorithm families.
  • On-policy vs off-policy: on-policy algorithms (like plain policy gradient) must discard all samples and collect new ones every time the policy changes even slightly; off-policy algorithms (like Q-learning) can reuse old samples, making them more sample efficient.
  • Efficiency spectrum: from least to most efficient, roughly: evolutionary/gradient-free methods, on-policy policy gradient, actor-critic (on- or off-policy), purely off-policy methods like Q-learning, then model-based RL.
  • Wall-clock time is different from sample efficiency: when simulation is cheap (for example, a fast chess simulator), the computationally expensive methods can be the sample-efficient ones, so the "less efficient" end of the spectrum can actually run faster in practice.
  • Convergence varies by family: policy gradient performs true gradient ascent on the objective; value-function fitting is a fixed-point iteration that minimizes Bellman error, not necessarily reward, and can diverge with neural networks; model-based RL's model training converges to an accurate model, but that doesn't guarantee a better policy.
  • Common assumptions: full observability (the Markov property), episodic (resettable) learning, and continuity or smoothness are all assumptions that different algorithm families rely on to varying degrees.

Before you watch

  • Understand the four algorithm families from the previous part of this lecture: policy gradient, value-based, actor-critic, and model-based.
  • Know the green-box/blue-box/orange-box anatomy of an RL algorithm used throughout this lecture.

Check your understanding

  1. What distinguishes an on-policy algorithm from an off-policy one, and why does that affect sample efficiency?
  2. Why might a very sample-inefficient algorithm still be the fastest choice in wall-clock time?
  3. Does policy gradient minimize Bellman error or perform gradient ascent on the true RL objective? What about value-function fitting?
  4. Name two assumptions common RL algorithms make about the environment, and one algorithm family associated with each.

Vocabulary

trade-off (noun)
A balance between two good things where you cannot fully have both.
Choosing an RL algorithm involves a trade-off between speed and stability.
sample efficiency (noun)
How much benefit an algorithm gets from a limited amount of collected data.
Sample efficiency measures how many samples are needed to learn well.
on-policy (adjective)
Describes an algorithm that can only learn from data collected by its current policy.
On-policy methods must discard old samples after each update.
off-policy (adjective)
Describes an algorithm that can learn from data collected by a different or older policy.
Off-policy methods can reuse old samples for training.
discard (verb)
To throw away or stop using something.
On-policy algorithms discard samples once the policy changes.
spectrum (noun)
A range of options between two extremes.
Algorithms sit at different points on the efficiency spectrum.
evolutionary method (noun)
An optimization approach inspired by natural selection, without gradients.
Evolutionary methods sit at the less sample-efficient end.
gradient-free (adjective)
Not using derivatives or gradients to guide optimization.
Some evolutionary methods are gradient-free.
wall-clock time (noun)
The actual real-world time a process takes to run.
Wall-clock time can differ from sample efficiency.
fixed-point iteration (noun)
A method that repeatedly applies a function until the result stops changing.
Value-function fitting is a form of fixed-point iteration.
Bellman error (noun)
The difference between a value estimate and what a consistency equation says it should be.
Value-function fitting minimizes the Bellman error.
diverge (verb)
To move further away from a stable or correct value instead of settling.
Value-function fitting can diverge with neural networks.
full observability (noun)
The assumption that an agent can see the complete true state of the environment.
Some algorithms assume full observability.
episodic (adjective)
Divided into separate, resettable episodes rather than running continuously.
Episodic learning allows the environment to reset.
convergence (noun)
The property of a process settling toward a stable final result.
Different algorithm families have different convergence guarantees.
guarantee (noun)
A mathematically proven promise about how a method behaves.
Policy gradient has a convergence guarantee that some methods lack.
assumption (noun)
Something taken to be true without direct proof, used as a basis for reasoning.
Full observability is a common assumption in RL algorithms.
continuity (noun)
The property of changing smoothly, without sudden jumps.
Some methods assume continuity in the environment's dynamics.
tune (verb)
To adjust a method's settings to make it work well.
Some algorithms are harder to tune than others.
resettable (adjective)
Able to be returned to a starting condition.
Episodic tasks are resettable after each run.

Chapters

← Lecture 4, Part 4: Types of RL Algorithms · Lecture 4, Part 6: Examples of Deep RL Algorithms →