Lecture 7, Part 2: Fitted Value and Fitted Q-Iteration

What this lecture covers

This part moves value-based methods beyond the tabular setting into function approximation. It first explains why storing a value function as a table is infeasible for large or continuous state spaces (the curse of dimensionality), then derives fitted value iteration, which fits a neural network to target values computed with the max-over-actions trick.

The key move is switching from a value function to a Q function: because a Q function recurrence conditions on both state and action, the sample (s, a, s') needed to fit it does not depend on the current policy, which removes the need for known transition dynamics or the ability to try multiple actions from the same state. This leads to fitted Q-iteration, the basis of most model-free value-based RL algorithms.

Key ideas

• Curse of dimensionality: a tabular value function needs one entry per state, which becomes infeasible or infinite for high-dimensional or continuous state spaces such as images.
• Fitted value iteration: fits a neural network V_phi(s) to target values computed as max_a [r(s,a) + gamma * E[V(s')]], but still requires known transition dynamics to compute the expectation and to try multiple actions from the same state.
• Switching to a Q function recurrence: because Q(s,a) conditions the next state on a fixed (s,a), samples (s, a, s') remain valid regardless of what the current policy is, removing the dependence on known dynamics.
• Fitted Q-iteration: alternates between computing target values y_i = r(s_i,a_i) + gamma * max_a' Q_phi(s_i', a') using sampled transitions, and regressing Q_phi onto those targets.
• Off-policy by construction: fitted Q-iteration works with samples gathered by any policy, since it never requires actions from the current greedy policy, unlike actor-critic.
• No convergence guarantees with neural networks: fitted Q-iteration is guaranteed to converge for tabular representations but not in general for nonlinear function approximators.
• Algorithm structure: collect transitions with some policy, compute target values from the previous Q function, then run some number of gradient steps to fit a new Q function, optionally repeating the fit step multiple times before collecting more data.

Before you watch

• Watch Part 1 of this lecture on policy and value iteration, since fitted Q-iteration is presented as its function-approximation, model-free extension.
• Recall the Q function and Bellman backup definitions from the actor-critic lecture.

Check your understanding

1. Why is a tabular value function infeasible for a high-dimensional or continuous state space?
2. Why does fitted value iteration still require knowing the transition dynamics?
3. What change makes the Q function recurrence usable with samples from any policy?
4. What are the two steps of fitted Q-iteration, and what does each compute?
5. Why does fitted Q-iteration have convergence guarantees in the tabular case but not with neural network function approximation?