Lecture 10, Part 4: Extending LQR to Stochastic and Nonlinear Systems

What this lecture covers

Having derived LQR for deterministic linear dynamics in Part 3, this part extends it in two directions: to stochastic (Gaussian) dynamics, and to nonlinear dynamics via a linearize-and-iterate approach. It connects the resulting algorithm, iterative LQR, to Newton's method and explains a line-search fix needed for it to work reliably in practice.

You'll come away understanding why Gaussian noise doesn't change the LQR control law, how iterative LQR (iLQR) locally linearizes nonlinear dynamics and cost around a current trajectory, and why a backtracking line search on the forward pass is needed for good convergence.

Key ideas

• Gaussian dynamics don't change the control law: if the stochastic dynamics are Gaussian with the same mean as the deterministic linear case, the optimal u_t = K_t x_t + k_t control law is unchanged, though the visited states become stochastic.
• LQR yields a closed-loop policy: in the stochastic case, K_t x_t + k_t acts as a time-varying linear controller rather than a single fixed action sequence.
• Taylor expansion for local linearization: nonlinear dynamics and cost can be approximated locally, around a current trajectory x_hat, u_hat, by a linear dynamics term (first derivative) and a quadratic cost term (up to second derivative).
• Iterative LQR (iLQR): repeatedly linearizes dynamics and cost around the current best trajectory, runs the LQR backward pass on the deviations delta_x, delta_u, then runs a forward pass with the true nonlinear dynamics to get an updated trajectory.
• Connection to Newton's method: iLQR is an approximation of Newton's method applied to the trajectory optimization problem; using the true second derivative of the dynamics instead gives differential dynamic programming (DDP).
• Line search via alpha: scaling the constant term k in the forward pass by a factor alpha between 0 and 1 lets iLQR backtrack toward the previous trajectory when a full step would make things worse, similar to trust-region ideas from policy gradient methods.

Walkthrough

Stochastic dynamics (0:00)

For linear dynamics with added Gaussian noise, the same LQR control law remains optimal because the symmetry of the Gaussian causes deviations to cancel out in the quadratic cost. The practical difference is that the states you visit are now themselves random, so LQR effectively becomes a closed-loop, time-varying linear policy rather than a single fixed plan.

Nonlinear dynamics and iterative LQR (3:08)

For nonlinear systems, the lecture introduces iterative LQR (also called differential dynamic programming or iLQG in the linear-Gaussian case). The idea is to use a Taylor expansion to approximate the nonlinear dynamics and cost locally around a current candidate trajectory, producing a linear-quadratic problem in the deviations delta_x and delta_u. Running standard LQR on this local problem, then applying the result with the true nonlinear dynamics in a forward pass, and updating the reference trajectory, repeats until convergence.

Newton's method connection (7:27)

The lecture draws a direct parallel between iLQR and Newton's method for minimizing a nonlinear function: both repeatedly form a local quadratic approximation and jump to its optimum. iLQR differs from full Newton's method by ignoring second derivatives of the dynamics; including them recovers DDP, at the cost of needing a 3D tensor for the second-order dynamics term.

Line search for stability (10:12)

Because a full Newton-style step can overshoot and land at a worse point, the lecture describes scaling the forward pass's constant term by alpha. Setting alpha to zero reproduces the previous trajectory exactly, and increasing it moves further toward the new solution, so searching over alpha (for example, reducing it until cost improves, or using a bracketing line search) makes iLQR far more reliable in practice.

Before you watch

• Watch Lecture 10, Part 3 first for the deterministic LQR derivation and the K_t, k_t notation this part builds on.
• A basic grasp of Newton's method and Taylor expansions helps with the nonlinear extension.

Check your understanding

1. Why does adding Gaussian noise to the dynamics leave the LQR control law unchanged, and what does change as a result?
2. What quantities does iLQR linearize or approximate at each iteration, and around what point?
3. Why does the forward pass in iLQR use the true nonlinear dynamics rather than the linearized approximation?
4. How is iLQR related to Newton's method, and what does it omit compared to full Newton's method?
5. What problem does the line-search parameter alpha solve, and what happens when alpha = 0?