Lecture 10, Part 3: Trajectory Optimization with the LQR

What this lecture covers

Building on the black-box planning methods from Part 2, this part of Lecture 10 turns to trajectory optimization methods that exploit derivatives of the dynamics and cost. It switches to optimal-control notation (x for state, u for action, cost instead of reward) and works toward deriving the linear quadratic regulator (LQR), a classic second-order shooting method.

After watching, you'll be able to explain why first-order gradient descent struggles on shooting-style trajectory objectives, distinguish shooting from co-location methods, and follow the backward-then-forward recursion that LQR uses to compute an optimal action sequence for linear dynamics with quadratic cost.

Key ideas

• Optimal-control notation: x_t/u_t and cost replace the s_t/a_t and reward used elsewhere in the course; the underlying math is the same up to a sign.
• Shooting methods: optimize only over actions, substituting the dynamics constraint directly into the objective; numerically unstable because early actions have an outsized effect on the final cost.
• Co-location methods: optimize over both states and actions subject to dynamics constraints, which tends to be better conditioned but is more complex to set up.
• Linear quadratic regulator (LQR): assumes linear dynamics (x_t = F_t [x_{t-1}; u_{t-1}] + f_t) and quadratic cost, and solves for the optimal action sequence in closed form.
• Backward recursion: starting from the last time step and working backward, LQR expresses each optimal action as u_t = K_t x_t + k_t and each value function as a quadratic form in x_t.
• Forward recursion: once K_t and k_t are known for every step, starting from the known initial state x_1, you compute u_1, then x_2 via the dynamics, then u_2, and so on to the end of the trajectory.
• Q-function and value function: in this setting the Q-function is the total cost from a given state-action pair to the end of the trajectory, and the value function is the total cost from a given state under the optimal policy.

Walkthrough

Shooting vs. co-location (4:03)

The lecture reframes model-based planning as an unconstrained optimization by substituting the dynamics constraint directly into the cost. Applying first-order gradient descent to this directly tends to work poorly because chaining many Jacobians together produces vanishing or exploding gradients, so second-order methods are preferred. Shooting methods optimize only actions and are sensitive to the first action's outsized effect on the trajectory; co-location methods optimize over states and actions jointly with explicit dynamics constraints, giving better numerical conditioning at the cost of more complexity.

LQR problem setup (7:15)

LQR assumes deterministic linear dynamics and a quadratic cost function, with potentially different linear/quadratic coefficients at each time step. The derivation starts at the last time step: since the final action u_T only affects the cost at time T, you can solve for the u_T that minimizes a quadratic expression in u_T and x_T by setting its derivative to zero, giving u_T = K_T x_T + k_T.

Deriving the value function and recursing backward (12:00, from 7:15 section)

Substituting the optimal u_T back into the objective produces the value function V(x_T), a quadratic form in x_T alone. This value function then feeds into the Q-function for the preceding time step, t-1, through the linear dynamics that relate x_t to x_{t-1} and u_{t-1}. The same derivative-and-solve procedure yields u_{t-1} = K_{t-1} x_{t-1} + k_{t-1}, and the pattern repeats all the way back to t=1.

Solving the LQR recursion (20:11)

Once the backward pass has produced K_t and k_t for every time step, the algorithm switches direction. Since x_1 is known, you compute u_1 = K_1 x_1 + k_1, then use the dynamics to get x_2, then u_2, and so on through x_T. The lecture describes this as "unzipping" the problem going backward and "zipping it back up" going forward, producing a full optimal trajectory for the linear-quadratic case. This sets up the extension to nonlinear dynamics covered later in the lecture.

Before you watch

• Watch Lecture 10, Part 2 first for the open-loop planning framing and why derivative-free methods have a dimensionality limit.
• Be comfortable with basic matrix calculus (taking derivatives of quadratic forms) and the chain rule, since the LQR derivation relies on both.

Check your understanding

1. Why does naive first-order gradient descent perform poorly on shooting-style trajectory optimization objectives?
2. What distinguishes a shooting method from a co-location method in trajectory optimization?
3. Why must the cost function be quadratic (not linear) for the LQR derivation to produce a useful solution?
4. Explain why LQR is solved with a backward pass followed by a forward pass rather than in one direction.
5. What do the matrix K_t and vector k_t represent in the LQR solution u_t = K_t x_t + k_t?