Lecture 23, Part 1: Challenges and Open Problems in Deep RL

What this lecture covers

This is the opening part of the final lecture of Berkeley CS285. It starts with a map of the course, showing how imitation learning, model-free RL (value-based, policy gradient, actor-critic), model-based RL, exploration, and control-as-inference relate to one another as branches of learning-based control.

The bulk of the lecture surveys open problems in deep RL. It covers core algorithmic challenges, stability and hyperparameter sensitivity across value-based, policy-gradient and model-based methods, then works through a rough hierarchy of sample efficiency across algorithm classes, and closes by discussing why generalization and scale remain much harder in RL than in supervised learning.

Key ideas

• Course map: learning-based control splits into imitation learning (from demonstrations) and RL (from rewards); RL splits into model-free (value-based, policy gradient, actor-critic) and model-based methods, with exploration, unsupervised RL objectives, and control-as-inference cutting across all of them.
• Stability and hyperparameter sensitivity: value-based methods with deep function approximation generally lack convergence guarantees, so techniques like target networks, replay buffers, and normalization are practical fixes for a deeper theoretical gap.
• The "magic" of deep learning is still unexplained: why supervised deep learning generalizes despite heavy overparameterization is not fully understood, and it is unclear whether the same implicit regularization helps value-based RL.
• Policy gradient trade-off: these methods largely avoid the bias of function approximation but suffer from high, sometimes catastrophically high, variance, making batch size, learning rate, and baseline design critical.
• Model-based RL's own challenges: more accurate models do not always yield better policies, since not all model errors matter equally, and policies can exploit model inaccuracies adversarially.
• Sample efficiency hierarchy: roughly ordered from least to most efficient: gradient-free methods, fully online on-policy methods (A2C/A3C), batch policy gradient methods (TRPO/PPO), off-policy methods with replay buffers, model-based RL, and shallow model-based methods like PILCO, with roughly an order-of-magnitude efficiency gain at each step.
• Efficiency isn't the only cost: less sample-efficient methods can still be preferable when they parallelize well or need less compute per update, since wall-clock time and compute cost also matter.
• Generalization gap versus supervised learning: supervised deep learning is evaluated on generalization across huge, diverse datasets, while RL is typically evaluated on mastery of narrow tasks, making breadth and generalization much less studied and much harder to achieve at scale.
• The RL "outer loop" problem: because RL learns through continuous interaction, changing the method often means re-running the entire interactive training process, which becomes impractical at internet scale, unlike supervised learning's fixed-dataset workflow.

Walkthrough

Course review and the map of learning-based control (0:00)

The lecture draws a conceptual map of the course: learning-based control covers imitation learning and reinforcement learning; RL splits into model-free methods (value-based, policy gradient, actor-critic) and model-based methods, with exploration, skill discovery, and control-as-inference as cross-cutting tools, and notes this is a rough overview rather than an exhaustive list of topics covered.

Core algorithmic challenges: stability and convergence (3:00)

The lecture distinguishes RL-specific challenges (stability, efficiency, generalization) from challenges rooted in RL's problem assumptions (such as needing a reward function). It explains why value-based methods with deep networks generally lack convergence guarantees, why many stabilizing tricks exist as practical fixes, and raises the open question of whether deep learning's mysterious generalization properties extend to value-based RL.

Policy gradient and model-based challenges (8:02)

Policy gradient methods are shown to trade bias for variance, which can become catastrophically large and unpredictable, requiring careful tuning of batch size and baselines. Model-based RL is described as effectively an adversarial process, where a policy can exploit small model inaccuracies, and where more accurate models don't always translate into better policies.

A hierarchy of sample efficiency (12:04)

The lecture ranks RL algorithm classes by sample efficiency, from gradient-free evolutionary methods through online policy gradient methods, batch policy gradient methods, off-policy replay-buffer methods, model-based RL, and shallow model-based methods, with concrete real-time equivalents (from roughly 15 days down to seconds) for a half-cheetah locomotion task, while noting less efficient methods can still win on parallelism or compute cost.

Generalization, scale, and the outer-loop problem (17:09)

The lecture contrasts supervised learning's emphasis on large-scale, diverse training evaluated by generalization with RL's typical emphasis on mastering narrow tasks, and explains why RL's continuous-interaction workflow makes the "outer loop" of retraining after every method change far more costly at scale than in supervised learning, illustrated with humanoid locomotion examples that generalize poorly beyond their training terrain.

Before you watch

• Complete or review the core CS285 sequence, including model-free RL, model-based RL, and exploration lectures, since this lecture assumes familiarity with all of them.
• Recall homework experiences with Q-learning, policy gradients, and model-based methods, which this lecture uses as concrete reference points for discussing stability and efficiency.

Check your understanding

1. Why do value-based deep RL methods generally lack convergence guarantees, and what practical techniques compensate for this?
2. What is the fundamental trade-off between policy gradient methods and value-based or model-based methods, in terms of bias and variance?
3. According to the sample-efficiency hierarchy, why might someone choose a less sample-efficient algorithm in practice?
4. Why does the lecture argue that RL's "outer loop" workflow becomes impractical at large scale, compared to supervised learning?