Lecture 10, Part 2: Stochastic Optimization for Planning

What this lecture covers

This part of CS 285's model-based planning lecture asks how to pick a good sequence of actions when you already have a dynamics model but want to treat the optimization as a black box. It follows directly from the open-loop planning setup introduced earlier in Lecture 10 and builds toward the derivative-based methods covered in the next part.

You'll come away able to describe random shooting and the cross-entropy method (CEM) for continuous, black-box control problems, and Monte Carlo tree search (MCTS) for discrete, stochastic problems such as board games and Atari.

Key ideas

• Open-loop planning: given a starting state, produce a full sequence of actions that maximizes reward, without replanning along the way.
• Random shooting: sample several action sequences from a distribution (e.g. uniform), evaluate each with the objective, and keep the best; simple and parallelizable but relies on luck.
• Cross-entropy method (CEM): iteratively sample actions, keep the top-scoring "elite" fraction (often the best 10 percent), refit the sampling distribution (typically a Gaussian) to the elites, and repeat.
• CMA-ES: a more sophisticated extension of CEM that adds momentum-like terms and can reach good solutions with smaller sample sizes.
• Dimensionality limit: these black-box methods tend to struggle past roughly 30 to 60 action dimensions, though correlated time steps (e.g. 10 dimensions over 15 steps) can extend that range.
• Monte Carlo tree search: a closed-loop, tree-based planning method well suited to discrete, stochastic settings such as board games and poker; it avoids the exponential cost of exhaustively expanding a search tree.
• UCT tree policy: a common MCTS rule that scores each node by its average value plus a bonus for nodes visited less often, balancing exploitation and exploration.

Walkthrough

Random shooting and its limits (0:00)

The lecture reframes planning as an unconstrained optimization over a concatenated action sequence A, ignoring any special temporal structure. The simplest approach, random shooting, samples several candidate sequences and picks the one with the highest objective value. It's trivial to implement and runs efficiently in parallel on modern hardware, but its quality depends entirely on getting lucky with the random samples.

Cross entropy method (4:00)

CEM improves on random shooting by iteratively refitting the sampling distribution toward the best-performing samples. Each round: sample actions from the current distribution, evaluate them, select the elite subset with the highest returns, and fit a new distribution (commonly a Gaussian) to those elites. With enough samples and a wide enough starting distribution, CEM can in principle find the global optimum, and CMA-ES extends the idea with momentum-style updates.

Cross entropy benefits and limits (8:00)

Both random shooting and CEM are fast when parallelized and don't require the dynamics to be differentiable, but they share a dimensionality ceiling: beyond roughly 30 to 60 dimensions the random sampling stops covering the space well. They also only produce open-loop plans, with no feedback during execution.

Monte Carlo tree search (14:57)

MCTS handles discrete, stochastic problems (Atari games, board games such as Go) by expanding a search tree selectively rather than exhaustively. At depth limits, a rollout with a baseline (even random) policy approximates the value of a node. The lecture works through a concrete example: expanding two candidate first actions, tracking accumulated return Q and visit count N at each node, and using the UCT rule — average value plus an exploration bonus that shrinks with visit count — to decide which branch to expand next. Once the computational budget runs out, the algorithm takes the best-scoring action from the root. The lecture notes that combining MCTS with a learned rollout policy and value function, as AlphaGo does, can substantially improve results.

Before you watch

• Be comfortable with the model-based planning setup from earlier in Lecture 10, including the idea of optimizing an action sequence against a known dynamics model.
• Familiarity with basic probability (sampling, fitting a Gaussian to data) helps with the cross-entropy method section.

Check your understanding

1. Why does random shooting scale well on modern hardware, and what is its main weakness?
2. Describe one full iteration of the cross-entropy method, including how the elite set is chosen and used.
3. Why is exhaustive tree search over actions and states exponential in the planning horizon, and how does MCTS avoid this?
4. In the UCT rule, why does a node's exploration bonus shrink as its visit count grows?
5. What kinds of problems is MCTS particularly well suited for, and why?