Lecture 10, Part 1: Introduction to Model-Based Planning

What this lecture covers

This opening part of Lecture 10 shifts the course from model-free methods to model-based reinforcement learning. Before covering how to learn a dynamics model, it asks a prior question: if you already know the transition dynamics of a system, how should you use that knowledge to choose actions? It surveys where known or easily-modeled dynamics arise in practice, formalizes planning and trajectory optimization as constrained optimization problems, and introduces the crucial distinction between open-loop and closed-loop planning.

This follows the first nine lectures on model-free methods (Q-learning and policy gradients), which deliberately avoided needing or learning a dynamics model, and sets up subsequent lectures on optimal control algorithms and learned dynamics models.

Key ideas

• Model-free versus model-based: prior lectures assumed the transition probabilities p(s_{t+1} | s_t, a_t) were unknown and never attempted to learn or use them; today's lecture assumes they are fully known.
• Where known dynamics come from: some domains (Atari, chess, Go) have dynamics specified by rules; some physical systems (a car's kinematics on a clean road) are easy to model by hand; simulators provide known dynamics as well; system identification fits unknown parameters of an otherwise known model structure.
• Optimal control, trajectory optimization, and planning: these overlapping terms describe selecting a sequence of states and actions to optimize an outcome; trajectory optimization typically uses smooth gradient-based optimization, planning often considers discrete branching possibilities, and optimal control is the general umbrella problem.
• Deterministic formulation: minimize the sum of costs (or maximize rewards) over a sequence of actions, subject to each successive state being exactly determined by the dynamics function applied to the previous state and action.
• Stochastic formulation: define a distribution over state sequences given an action sequence and maximize expected reward under it, but this "open-loop" formulation can be badly suboptimal.
• Open-loop versus closed-loop planning: open-loop planning commits to a full action sequence in advance without observing future states; closed-loop planning (a policy) lets the agent observe each new state before choosing the next action, which matters whenever new information will be revealed over time.
• Local versus global policies: reinforcement learning typically learns expressive global policies (like neural networks), but many optimal-control applications use much simpler local policies, such as a time-varying linear feedback controller around a planned trajectory.

Walkthrough

From model-free to model-based RL (1:07)

The lecture recaps the reinforcement learning objective and trajectory distribution used throughout the course, and highlights that all prior algorithms managed to avoid ever needing to know or predict transition probabilities directly. Today's question is the reverse: what becomes possible if you do know the dynamics?

Where known dynamics come from (3:45)

The lecture lists settings where dynamics are known outright (games with fixed rules), easy to model by hand (simple physical kinematics), available via simulation, or partially known with unknown parameters that can be fit through system identification. It notes that knowing the dynamics opens up a range of powerful algorithms unavailable in the model-free setting.

Formulating planning as optimization (6:41)

With known dynamics but no policy, the problem becomes selecting an action sequence directly. The lecture distinguishes trajectory optimization, planning, and optimal control as closely related framings of this problem, then writes the deterministic case as a constrained optimization: minimize total cost over an action sequence subject to the state transitions given by the (known) dynamics function.

Deterministic versus stochastic planning (11:56)

In the deterministic case, an agent observes the current state and computes an optimal action sequence in one shot. Extending this naively to stochastic dynamics, by maximizing expected reward over a distribution of state sequences given a fixed action sequence, can be badly suboptimal whenever the environment will reveal useful information partway through, illustrated with the example of committing to exam answers before seeing the questions.

Open-loop versus closed-loop planning (15:23)

The lecture defines open-loop planning as committing to a full action sequence without observing intermediate states, and closed-loop planning as choosing each action based on the newly observed state, closing the loop between perception and control. Open-loop planning can be optimal in simple deterministic settings but is generally suboptimal in stochastic ones.

Closed-loop policies, global and local (16:45)

Closed-loop planning amounts to choosing a policy pi(a_t | s_t), with the same objective as standard reinforcement learning. The lecture notes that policies don't need to be as expressive as neural networks: many optimal control applications, like guiding a rocket's trajectory, use simple local policies such as time-varying linear feedback controllers that correct small deviations from a planned trajectory.

Before you watch

• Be comfortable with the reinforcement learning objective and trajectory distribution notation used throughout the course, since the lecture recaps but does not re-derive them.
• Recall that Q-learning was framed earlier specifically to avoid needing transition probabilities; this lecture explicitly revisits that choice.

Check your understanding

1. What are three different practical settings where transition dynamics are known or easy to obtain?
2. How does the deterministic planning optimization problem differ in formulation from the stochastic one?
3. Why can naive stochastic open-loop planning be badly suboptimal, and what does the exam example illustrate about this?
4. What is the difference between a global policy and a local, time-varying linear policy, and when might the latter be preferable?