Lecture 11, Part 2: Why Model-Based RL Underperforms, and Uncertainty

What this lecture covers

Continuing directly from Part 1's model-based RL version 1.5 (MPC), this part examines a real experimental result showing that iterative model-based RL, despite fixing distributional shift, still substantially underperforms model-free RL. It then introduces the idea of uncertainty-aware planning as a partial remedy, setting up the uncertainty-estimation methods covered in the next part.

After watching, you'll be able to explain why high-capacity models struggle in the low-data regime that early model-based RL training requires, why planners exploit overfit model errors, and how reasoning about model uncertainty (not just the mean prediction) can produce more cautious, sensible early-training behavior.

Key ideas

• Empirical performance gap: on the HalfCheetah task, a deep model-based RL learner (version 1.5) reached around 500 reward, far below the roughly 5,000 a model-free learner achieved with much more data.
• Low-data regime problem: neural network models need to be reasonably accurate even early in training, when data is scarce, but high-capacity models are best suited to the high-data regime, so they underperform early on.
• Planner exploits model errors: when the model overfits and has spurious optimistic peaks, the planner will preferentially select trajectories that land on those peaks, worsening the effective error.
• Uncertainty-aware planning: instead of predicting a single next state, predict a distribution over possible next states (reflecting uncertainty about the model itself, not environment noise) and plan using the expected reward under that uncertainty.
• Cliff-edge intuition: with high model uncertainty near a goal at a cliff edge, taking the expected reward under uncertainty naturally makes the planner keep a safe distance, without explicitly being told to be cautious.
• Caveats: uncertainty-aware planning using expected value is neither pessimistic (robust/worst-case) nor optimistic (exploration-seeking); overly cautious uncertainty modeling can also hamper exploration.

Walkthrough

The performance gap in model-based RL (0:24)

The lecture presents Berkeley experiments on HalfCheetah comparing model-based RL (version 1.5, replanning every step) to a model-free learner bootstrapped from the same setup. The model-based approach reaches modest but non-trivial performance quickly, while the model-free learner eventually reaches much higher reward given far more training time.

Why the gap exists (2:11)

The core issue is an overfitting problem compounded by distributional shift: neural network models must avoid overfitting on the small early datasets while still having enough capacity to fit well later, which is a difficult balance. When the model has small spurious peaks of overestimated reward, the planner actively seeks out and exploits those peaks, since it's searching for the best predicted outcome.

How uncertainty estimation helps (4:15)

Rather than predicting a single next state, the model predicts a distribution over possible next states, capturing uncertainty about which dynamics model is correct given the data so far. The cliff-edge analogy illustrates that planning with the expected reward under this uncertainty naturally produces cautious behavior near risky, poorly-understood regions, without an explicit pessimism term — because the expected value already accounts for the chance of a bad outcome.

Caveats around uncertainty-aware planning (7:10)

The lecture is careful to distinguish expected-value planning under uncertainty from both pessimistic (worst-case, robust) approaches and optimistic (exploration-seeking) approaches — each is a legitimate but different strategy, with the course returning to optimism-driven exploration in later lectures. It also flags that overly cautious uncertainty estimates can themselves suppress the exploration needed to eventually reduce that uncertainty.

Before you watch

• Watch Lecture 11, Part 1 first for the model-based RL version 1.0/1.5 algorithms and the distributional shift problem this part measures empirically.
• Recall the HalfCheetah task from Homework 1, referenced directly as the experimental setting.

Check your understanding

1. What does the HalfCheetah experiment show about model-based versus model-free RL performance, and over what timescale?
2. Why do high-capacity neural network models specifically struggle in the low-data regime of early model-based RL training?
3. Explain how a planner can "exploit" errors in an overfit dynamics model.
4. How does predicting a distribution over next states, rather than a single value, lead to more cautious planning near risky regions?
5. Why is expected-value planning under uncertainty different from both pessimistic and optimistic planning strategies?