Lecture 10, Part 5: A Case Study in Optimal Control

What this lecture covers

This short closing part of Lecture 10 is a case study rather than new theory: it shows what iterative LQR (from Part 4) can achieve when combined with model predictive control (MPC) and known dynamics, using the paper "Synthesis and stabilization of complex behaviors through online trajectory optimization" by Tassa, Erez, and Todorov. It sits at the end of the optimal-control segment of the course and motivates the shift to learned models in the following lectures.

After watching, you'll understand the basic idea of model predictive control, and you'll have seen concrete examples of iLQR-based controllers discovering complex behaviors automatically, without any learning, purely from known system dynamics.

Key ideas

• Model predictive control (MPC): at every time step, observe the current state, plan a full action sequence, execute only the first action, then replan from the newly observed state.
• No learning required: the demonstrated behaviors are discovered entirely by online optimization using known dynamics, not by training a policy on data.
• Robustness to perturbations: because MPC replans constantly, the controllers recover from external pushes and even tolerate dynamics that are somewhat misspecified (for example, incorrect mass).
• Short-horizon planning needs careful costs: for the more complex 3D humanoid, the controller only plans a short horizon ahead, so the cost function has to be heavily engineered to get sensible behavior.
• Limits of known-dynamics control: this approach works well for systems with easily modeled dynamics (like a car), but is impractical for tasks such as folding a towel or sorting objects, where accurate dynamics models are hard to obtain, motivating learned models in later lectures.

Walkthrough

Introduction to the case study (0:00)

The lecture introduces model predictive control as replanning at every step with iLQR as the inner-loop optimizer, based on the Tassa, Erez, and Todorov paper, and frames the video demo as evidence that classical optimal control with known dynamics can produce impressively capable behavior.

Demo across several systems (1:50)

The video walkthrough covers several simulated systems: a two-degree-of-freedom acrobot that automatically discovers a swing-up motion and recovers from perturbations; a swimming snake that discovers an undulating gait to reach a target while avoiding an obstacle; and a hopper that stands up on its own and stays upright under perturbations, including cases where the controller is given incorrect mass values.

Humanoid control (4:42)

A more complex 3D humanoid example is shown standing up and performing rudimentary stepping and balancing, reacting reasonably to strong perturbations even though its planning horizon is short and its cost function required substantial hand-engineering.

Further readings and next steps (5:21)

The lecture points to the original differential dynamic programming monograph by Mayne and Jacobson, the MPC paper just discussed, and a paper on probabilistic and trust-region alternatives to deterministic line search for LQR. It closes by noting that known-dynamics control breaks down for tasks with dynamics that are hard to model by hand, setting up the following lectures on learning dynamics models.

Before you watch

• Watch Lecture 10, Parts 3 and 4 first, since this case study assumes familiarity with LQR and iterative LQR.
• No new mathematical background is needed beyond what earlier parts of Lecture 10 covered.

Check your understanding

1. What is the core loop of model predictive control, and how does it differ from computing one open-loop action sequence?
2. Why were the demonstrated behaviors (swing-up, swimming gait, standing) described as discovered automatically rather than learned?
3. Why does MPC's constant replanning make the controller somewhat robust to incorrect dynamics assumptions?
4. Why did the humanoid example require a heavily engineered cost function?
5. What kinds of real-world tasks does the lecture suggest are poorly suited to known-dynamics optimal control, and why?