Lecture 18: Continuous-State MDPs and Fitted Value Iteration

What this lecture covers

This lecture extends the MDP and value-iteration machinery from the previous two lectures, which assumed a small number of discrete states, to problems with continuous state spaces such as driving a car, flying a helicopter, or balancing an inverted pendulum. It answers two connected questions: how do you represent a continuous state space at all, and how do you approximate the value function over it without enumerating every state?

The lecture first covers discretization and explains why it breaks down as the number of state dimensions grows (the curse of dimensionality), then introduces model-based reinforcement learning, where a simulator of the robot's dynamics, built either from physics or learned from data, stands in for the true state transition probabilities. It ends by deriving fitted value iteration, an algorithm that approximates the value function as a linear function of state features and fits it using sampled transitions from the simulator, and discusses how to turn the learned value function into a real-time controller.

Key ideas

• Continuous state representation: robots such as cars, helicopters, and an inverted pendulum are naturally modeled with continuous state vectors in R^n (position, orientation, and their rates of change), rather than a small number of discrete states.
• Discretization: laying a grid over a continuous state space turns it back into a discrete MDP that value or policy iteration can solve directly, and works fine for low-dimensional problems.
• Curse of dimensionality: discretizing each of n dimensions into k values produces k^n discrete states, so discretization becomes impractical as dimensionality grows past roughly six to eight.
• Model (simulator): a function that takes a state and action and returns a next state, built either from physics equations or software, or learned from recorded trajectories using supervised learning (often linear regression).
• Model-based versus model-free RL: model-based RL trains a policy in a learned or physics-based simulator before deploying it on the real system, which is preferred in robotics because real robots cannot afford to fail thousands of times; model-free RL, which learns directly from interaction, is more common where a perfect simulator (like a video game) already exists.
• Stochastic simulators: adding noise to a simulator during training makes the learned policy more robust to the simulator's inevitable inaccuracies and more likely to transfer to the real system.
• Fitted value iteration: approximates V*(s) as theta^T * phi(s), a linear function of hand-chosen state features, and fits theta with regression against target values computed by sampling the simulator at a fixed set of sampled states.
• Real-time deployment: once trained, a controller can drop the simulator's noise and, for each possible action, simulate the resulting next state and pick the action whose next state has the highest learned value, fast enough to run many times per second.

Walkthrough

Recap and modeling continuous state spaces (3:12)

The lecture briefly reviews the MDP notation, Bellman's equation, and value iteration from the previous two lectures, then turns to the question of how to represent state for a self-driving car, a helicopter, or an inverted pendulum. A car's state might be position, orientation, and their velocities (six numbers); a helicopter adds roll, pitch, and yaw for a twelve-dimensional state; and the classic inverted-pendulum problem uses a four-dimensional state (cart position and pole angle, plus their velocities). All of these are treated as vectors in R^n.

Discretization and the curse of dimensionality (10:22)

Discretization lays a grid over the continuous state space and treats each cell as one discrete state, reducing the problem back to ordinary value or policy iteration. The lecture compares this to fitting housing prices with a staircase function instead of a smooth curve: it is a crude approximation of the value function. The bigger problem is the curse of dimensionality: discretizing each of n dimensions into k values yields k^n states, so the state count explodes exponentially as dimensionality grows, which also affects very large discrete-state problems like factories with many machines.

Guidelines and the shift to function approximation (17:38)

The lecture offers rules of thumb: discretization works well for two- or three-dimensional state spaces, is workable with care (such as finer discretization on more sensitive dimensions) up to about six dimensions, and becomes impractical beyond seven or eight. As an alternative, it introduces the idea of approximating V* directly as a linear function of features of the state, theta^T * phi(s), drawing an explicit analogy to using linear regression with engineered features to predict housing prices.

Building a model: physics and learning from data (29:56)

Fitted value iteration needs a model, or simulator, of the robot: a function mapping a state and action to the next state. One way to build it is from physics equations or an open-source physics engine. The other, used more often in practice, is to record trajectories of a human pilot flying a real system and fit a supervised learning model, often linear regression of the form s_(t+1) = A*s_t + B*a_t, to predict the next state from the current state and action; nonlinear features can be added for a better fit.

Model-based RL and the importance of simulator noise (39:15)

The lecture distinguishes model-based RL, where a policy is trained in a simulator and then deployed on the real robot, from model-free RL, which learns directly on the system; model-based RL dominates robotics because real robots cannot tolerate the many failures learning requires, while model-free approaches suit domains with perfect, cheap simulators like video games. A key practical lesson is to make the simulator stochastic by injecting noise into predicted next states during training, since a deterministic simulator tends to produce policies that look good in simulation but transfer poorly to a real, imperfectly modeled robot.

Deriving fitted value iteration (49:28)

The algorithm samples a fixed set of states, then, for each state and action, draws samples of the next state from the simulator to estimate the expected future value, taking the max over actions to form a target value y_i for that state. Because V(s) is now a linear function of features rather than an independent value per state, the final step of each iteration runs linear regression to choose parameters theta that make theta^T * phi(s_i) close to each y_i, rather than directly overwriting values as in tabular value iteration. Feature choice for phi(s) works like feature engineering for housing prices, and since a simulator can generate effectively unlimited data, overfitting from using many features is less of a concern than in ordinary supervised learning.

From value function to a real-time policy (1:10:46)

Because the learned value function only implicitly defines the optimal policy, computing an action at deployment time means evaluating, for each possible action, what next state the simulator predicts and picking the action with the highest predicted value. For real-time control, the lecture recommends removing noise from the simulator and using a single sample (k=1) rather than a random number generator inside the control loop, and notes this can run fast enough to control a helicopter or car many times per second, with actual sensor readings replacing the simulated state at each step.

Before you watch

• Review Bellman's equation, value iteration, and policy iteration from the two previous lectures in this course, since this lecture builds directly on that notation.
• Be comfortable with linear regression, including fitting parameters by minimizing squared error.
• Recall the earlier discussion of discrete-state MDPs and the 11-state grid-world example, which this lecture contrasts with continuous state spaces.

Check your understanding

1. Why does the curse of dimensionality make discretization impractical for state spaces with more than about eight dimensions?
2. What are the two main ways to build a model (simulator) of a robot's dynamics described in the lecture?
3. Why does adding noise to a simulator during training tend to produce policies that work better on the real robot?
4. How does the update step in fitted value iteration differ from the update step in tabular value iteration?
5. Why is it preferable to remove randomness from the simulator and use a single sample when deploying a real-time controller on a physical robot?