Lecture 12, Part 1: Model-Based RL with Policies

What this lecture covers

Lecture 12 opens the final model-based RL topic in CS 285: using learned models to train global policies rather than just planning open-loop action sequences. It starts by reviewing the model-based RL version 1.5 (MPC) algorithm from Lecture 11 and its core limitation, then walks through a naive attempt at policy learning through backpropagation and explains carefully why that attempt tends to fail.

After watching, you'll be able to explain why open-loop and MPC-style planning are fundamentally sub-optimal, describe the "model-based RL version 2.0" computation-graph approach of backpropagating rewards through a learned policy and dynamics model, and articulate why this naive approach suffers from the same ill-conditioning and vanishing/exploding gradient problems as naively trained recurrent networks.

Key ideas

• Open loop is sub-optimal: committing to a sequence of actions in advance (even when replanned every step, as in MPC) cannot represent strategies that depend on information not yet observed.
• The math-test example: an agent choosing whether to take a test and then answer it cannot behave optimally open-loop, because it must commit to an answer before seeing the question; a closed-loop policy can wait to observe the state before deciding.
• Closed-loop policies needed: a policy pi(a|s) that reacts to whatever state is observed solves this, and is the objective of standard RL; the model-based twist is that p(s_t+1 | s_t, a_t) is explicitly modeled.
• Local vs. global closed-loop control: methods like iterative LQR do produce a feedback controller, but only locally valid near the planned trajectory; this lecture targets globally valid policies like neural networks.
• Model-based RL version 2.0 (naive): build a computation graph chaining policy, dynamics, and reward nodes across time steps, and directly backpropagate the total reward into the policy parameters.
• Why it fails: gradients from later rewards back to earlier policy parameters multiply many Jacobians together, causing vanishing or exploding gradients and severe ill-conditioning, the same phenomenon that afflicts naively trained recurrent networks and shooting-method trajectory optimization.
• No easy fix: unlike RNNs, where architectures like LSTMs can be designed to have well-behaved gradients, here the dynamics function must match the real physical dynamics, so its conditioning cannot be chosen for convenience.

Walkthrough

Recap and the limits of MPC (0:00)

The lecture recaps model-based RL version 1.5 (collect data, train a dynamics model, plan and replan every step via MPC), and highlights its central shortcoming: nearly all the planning methods covered so far, including MPC, are fundamentally open-loop, which is provably sub-optimal.

The math-test example and the case for closed-loop control (1:21)

Revisiting the math-test example from earlier in the course, the lecture shows that an open-loop strategy must commit to an answer before seeing the question, forcing a rational open-loop planner to skip the test entirely — even though a closed-loop strategy could safely take the test and answer correctly once it observes the question. The lecture notes MPC's constant replanning doesn't fix this because each individual plan is still computed open-loop, without accounting for future information gain.

Computation graph and model-based RL version 2.0 (5:39)

As a first, naive attempt at learning global closed-loop policies, the lecture proposes chaining learned policy, dynamics, and reward networks into one differentiable computation graph, then directly backpropagating the negated total reward to update the policy parameters — essentially applying standard deep learning optimization to the whole RL objective. This is dubbed "model-based RL version 2.0": collect data, train a dynamics model, backpropagate through the computation graph to optimize the policy, execute it, and retrain.

Why version 2.0 doesn't work well (13:57, preceded by discussion from roughly 8:30)

The lecture explains that gradients with respect to early actions are much larger than those for late actions — mirroring the sensitivity problem seen in shooting-method trajectory optimization — producing ill-conditioned optimization that first-order gradient descent handles poorly. Unlike trajectory optimization, second-order methods like LQR cannot be applied conveniently here because policy parameters couple all time steps together. The lecture draws a direct parallel to vanishing and exploding gradients in naively trained recurrent neural networks, but notes that fixes used for RNNs (like LSTM-style architectures with well-behaved dynamics) aren't available here, since the dynamics function must match reality rather than being chosen for favorable gradients. The lecture closes by previewing the actual solution used in practice: using learned models to generate synthetic data for otherwise-standard model-free RL algorithms.

Before you watch

• Watch Lecture 11 in full first, especially the model-based RL version 1.5 (MPC) algorithm and the math-test open-loop example introduced there.
• Familiarity with how vanishing and exploding gradients arise in recurrent neural networks is helpful for the final section.

Check your understanding

1. Why is the math-test example a case where open-loop planning is provably sub-optimal, even with replanning?
2. What distinguishes the closed-loop feedback controller produced by iterative LQR from a globally valid closed-loop policy like a neural network?
3. Describe the computation graph used in "model-based RL version 2.0" and what it backpropagates through.
4. Why do gradients with respect to early actions tend to be much larger than gradients with respect to late actions in this setup?
5. Why can't the fixes developed for recurrent neural network training (like LSTM architectures) be directly applied to fix model-based RL version 2.0?