Lecture 20: RL Debugging, Diagnostics, and Policy Search

What this lecture covers

This is the final lecture of Stanford's CS229. It answers a practical question left over from earlier in the course: when a reinforcement learning system flies (or drives, or trades) worse than expected, how do you figure out whether the problem is your simulator, your reward function, or your learning algorithm? Ng revisits the autonomous helicopter project to lay out a systematic diagnostic process, then introduces a new family of methods, direct policy search, that skips value function estimation and optimizes a policy directly.

By the end you can describe the three-way diagnostic for a poorly performing RL system, explain why and how a stochastic policy is parameterized and searched over, and follow the derivation showing that the REINFORCE update is, in expectation, a gradient ascent step. The lecture closes with guidance on when to prefer policy search over value-function methods, and a survey of where RL is applied in practice.

Key ideas

• Diagnostic order for RL systems: compare the learned policy's performance in simulation versus real life to check the simulator, then compare the learned policy's payoff against a human expert's to decide whether to improve the reinforcement learning algorithm or the reward function.
• Direct policy search: instead of estimating a value function and deriving a policy from it, you choose a parametric form for the policy pi_theta and optimize its parameters directly to maximize expected reward.
• Stochastic policy: a policy that outputs a probability distribution over actions (for example, a sigmoid of theta^T * state giving the chance of accelerating right) rather than a single deterministic action.
• REINFORCE algorithm: run the current stochastic policy for a trajectory, compute the total payoff, then update theta by a rule that, on expectation, points in the direction of the gradient of expected payoff, even though each individual update is noisy.
• Partially observable MDPs (POMDPs): when you cannot observe the full state (only noisy partial measurements), value-function methods become very hard, but direct policy search still works because it can be applied straight to the observations.
• Policy search versus value functions: policy search tends to work well for low-level control tasks (instinctive, seat-of-the-pants control like flying or steering) where a simple policy is adequate, while value-function approaches tend to work better for tasks needing multi-step reasoning, such as chess or Go.
• REINFORCE is inefficient: its gradient estimates are unbiased but high-variance, so it often needs a very large number of iterations and a small learning rate to converge.

Walkthrough

Debugging a reinforcement learning system (0:03)

The lecture opens with footage of the Stanford autonomous helicopter performing aerobatic stunts, then returns to a debugging example promised in an earlier lecture. Ng lays out the standard workflow: build a simulator, choose a reward function, run an RL algorithm to get a policy, and compare it against a human pilot. He explains that a learned policy failing to match human performance can stem from one of three causes: an inaccurate simulator, an RL algorithm that isn't maximizing reward well, or a reward function that doesn't capture what actually matters. The diagnostic is to first check whether the policy performs well in simulation; if not, the simulator is the bottleneck. If it performs well in simulation but not in reality, compare its payoff against a human pilot's payoff on the same reward function: if the learned policy scores lower, work on the RL algorithm; if it scores higher but still flies worse, work on the reward function. Ng notes that in real projects, the bottleneck shifts between these three areas over months, and that teams who follow this diagnostic process tend to be more efficient than those working by gut feeling.

Value function approximation in robotics (16:30)

Ng shows a video of a legged robot climbing over rough terrain using a learned value function that estimates the cost of placing a foot at different points on the terrain, based on a height map. He uses this as an example of fitted value iteration applied to a real robotics problem, and predicts that robotics, rather than game-playing, will be one of the most significant growth areas for reinforcement learning in coming years.

Direct policy search and the inverted pendulum (18:47)

Ng introduces policy search (also called direct policy search) as an alternative to the value-function approach used throughout the course: instead of estimating a value function and deriving a policy from it, you choose a functional form for the policy and optimize its parameters directly. Using the inverted pendulum as the running example, he defines a stochastic policy as a sigmoid function of the state (position, velocity, angle, and angular velocity) parameterized by theta, where the output gives the probability of accelerating right versus left. He walks through how different parameter settings change the policy's behavior, for example weighting the pole's angle versus the cart's position, and states the optimization goal: choose theta to maximize the expected sum of rewards from a fixed initial state, using a finite-horizon formulation.

Deriving the REINFORCE algorithm (34:25)

Ng writes out the REINFORCE algorithm: run the policy for a trajectory, compute the total payoff, and update theta by adding the learning rate times a term involving the payoff and the gradient of the log-probability of the actions taken. He then derives, step by step using the product rule from calculus and an algebraic trick of multiplying and dividing by the same terms, that this update is, in expectation, exactly equal to the gradient of the expected payoff with respect to theta. This is what makes REINFORCE a stochastic gradient ascent algorithm even though any single update is noisy, since it depends on a random trajectory. He notes this derivation holds for any differentiable stochastic policy, not just the sigmoid form shown, and that a continuous, differentiable policy is required for the gradient to be defined, unlike a hard left-or-right decision rule. He also briefly contrasts policy search with supervised learning from human demonstrations, and shows how a Gaussian-noise policy extends the approach to continuous actions such as steering angle.

Partially observable MDPs and when to prefer policy search (52:24)

Ng identifies two settings where direct policy search has a clear advantage over value-function methods. The first is partially observable MDPs (POMDPs), where you only get noisy, partial measurements of the state, for example an angle sensor and position sensor on the inverted pendulum without direct velocity readings. Value-function approaches generalize poorly to this setting, but policy search can be applied directly to the noisy observations (or to features derived from them, such as a Kalman filter's state estimate) with no change to the framework. The second consideration is whether the optimal policy or the optimal value function is simpler to represent: low-level, instinctive control tasks like flying or steering tend to have simple near-optimal policies, favoring policy search, while tasks requiring multi-step reasoning, like chess or Go, tend to favor value-function approaches. He notes that AlphaGo combined ideas from both families of methods.

Applications and closing thoughts (1:03:32)

Ng surveys where reinforcement learning is applied beyond game playing: a growing number of robotics applications, factory deployment optimization, early research on chatbots and AI guidance counselors, sequential medical treatment planning, and stock trading, where RL is used to break large trades into smaller ones to avoid moving the market price. The lecture closes with a review of the topics covered across the course, from supervised learning through unsupervised learning to reinforcement learning, and a personal message from Ng encouraging students to use their skills for meaningful, privacy-respecting work, followed by thanks to the class.

Before you watch

• This lecture assumes you already understand value iteration, policy iteration, and fitted value iteration for continuous-state MDPs from earlier lectures in this course.
• Familiarity with the finite-horizon MDP formulation and the idea of a reward function will make the direct policy search section easier to follow.
• Basic comfort with the product rule from calculus is useful for following the REINFORCE gradient derivation.

Check your understanding

1. A team's helicopter policy flies well in simulation but poorly in real life. According to the diagnostic in this lecture, what should they work on next, and why?
2. Why does direct policy search require a stochastic rather than a deterministic policy?
3. Explain, in your own words, why the REINFORCE update is described as a stochastic gradient ascent step even though each individual update is noisy.
4. Why do value-function approaches struggle in partially observable MDPs, and how does direct policy search avoid that problem?
5. Give an example of a low-level control task and a task requiring multi-step reasoning, and explain why each favors a different family of RL methods.