Lecture 7, Part 4: Why Fitted Q-Iteration Doesn't Converge

What this lecture covers

This closing part of the value-based methods lecture gives the theoretical explanation for a claim made earlier: fitted value iteration and fitted Q-iteration are not guaranteed to converge when using neural networks, even though tabular value iteration is. It builds this argument using the Bellman backup operator and a projection operator from supervised learning.

You come away understanding why each operator individually is a contraction, but why their composition is not, and why Q-learning's update does not correspond to gradient descent on any well-defined objective.

Key ideas

• Bellman backup operator (B): applies the value iteration update to a whole value function vector at once; it is a contraction in the infinity norm, meaning it always moves any two value functions closer together by a factor of gamma.
• Fixed point: the optimal value function V* is the unique fixed point of B, so repeatedly applying B to any starting vector converges to V* in the tabular case.
• Projection operator (Pi): represents the supervised learning step of fitted value iteration, projecting the backed-up value onto the space of representable neural network value functions; it is a contraction in the L2 norm.
• Why fitted value iteration can fail to converge: B contracts in the infinity norm and Pi contracts in the L2 norm, but the composition Pi * B is not guaranteed to be a contraction under any norm, so each fitted value iteration step can move away from the optimum even though both operators individually contract.
• Same argument applies to fitted Q-iteration and Q-learning: the analogous Bellman operator with a max over actions is still a contraction in the infinity norm, but combined with a projection step, convergence is not guaranteed with function approximation.
• Q-learning is not gradient descent: because the target value itself depends on the current Q function but the gradient is not taken through that dependence, the update does not follow the gradient of a well-defined objective; taking the true gradient (residual gradient) is possible but tends to have poor numerical behavior in practice.
• Actor-critic has the same issue: fitted bootstrapped policy evaluation in actor-critic is subject to the same non-convergence argument, since it also combines a Bellman backup with a projection.

Before you watch

• Watch Parts 1 through 3 of this lecture on policy iteration, fitted Q-iteration, and Q-learning, since this part explains why those algorithms lack convergence guarantees.

Check your understanding

1. Why is the Bellman backup operator B a contraction, and in which norm?
2. Why is the projection operator Pi a contraction in the L2 norm?
3. Why does composing two contractions, B and Pi, not guarantee that Pi * B is a contraction?
4. Why is the Q-learning update not the gradient of a well-defined objective?
5. Why does this non-convergence argument also apply to actor-critic's fitted value function updates?