Lecture 4, Part 5: Comparing RL Algorithms

What this lecture covers

This part steps back from specific algorithms to ask why the field has so many of them instead of one best method. It answers that question by walking through the trade-offs that push you toward one algorithm family or another: how many environment samples an algorithm needs, how stable and easy to tune it is, whether it is guaranteed to converge, and what assumptions it makes about the environment.

After watching, you can explain the difference between on-policy and off-policy algorithms, place different algorithm families on a sample-efficiency spectrum, and describe which algorithm types have convergence guarantees and which do not.

Key ideas

• Sample efficiency: how many environment samples an algorithm needs before it learns a good policy; this is the main axis separating algorithm families.
• On-policy vs off-policy: on-policy algorithms (like plain policy gradient) must discard all samples and collect new ones every time the policy changes even slightly; off-policy algorithms (like Q-learning) can reuse old samples, making them more sample efficient.
• Efficiency spectrum: from least to most efficient, roughly: evolutionary/gradient-free methods, on-policy policy gradient, actor-critic (on- or off-policy), purely off-policy methods like Q-learning, then model-based RL.
• Wall-clock time is different from sample efficiency: when simulation is cheap (for example, a fast chess simulator), the computationally expensive methods can be the sample-efficient ones, so the "less efficient" end of the spectrum can actually run faster in practice.
• Convergence varies by family: policy gradient performs true gradient ascent on the objective; value-function fitting is a fixed-point iteration that minimizes Bellman error, not necessarily reward, and can diverge with neural networks; model-based RL's model training converges to an accurate model, but that doesn't guarantee a better policy.
• Common assumptions: full observability (the Markov property), episodic (resettable) learning, and continuity or smoothness are all assumptions that different algorithm families rely on to varying degrees.

Before you watch

• Understand the four algorithm families from the previous part of this lecture: policy gradient, value-based, actor-critic, and model-based.
• Know the green-box/blue-box/orange-box anatomy of an RL algorithm used throughout this lecture.

Check your understanding

1. What distinguishes an on-policy algorithm from an off-policy one, and why does that affect sample efficiency?
2. Why might a very sample-inefficient algorithm still be the fastest choice in wall-clock time?
3. Does policy gradient minimize Bellman error or perform gradient ascent on the true RL objective? What about value-function fitting?
4. Name two assumptions common RL algorithms make about the environment, and one algorithm family associated with each.