Lecture 17, Part 2: RL Theory

What this lecture covers

This part continues the CS285 theory lecture, moving from the model-based analysis of part 1 to an idealized version of fitted Q-iteration, a model-free method. Because real fitted Q-iteration is not guaranteed to converge, the lecture studies a simplified version amenable to analysis, and asks how far the resulting q_hat_k ends up from q_star after many iterations.

You'll see the error decomposed into two separate sources: sampling error, from using an empirical Bellman operator t_hat built from averaged rewards and transitions instead of the true operator t, and approximation error, from fitting the Q-function inexactly at each iteration. The lecture then combines both to produce a bound on the asymptotic error of approximate fitted Q-iteration.

Key ideas

• Idealized fitted Q-iteration: models the effect of averaging many sample losses for the same state-action pair as a backup under an empirical transition model p_hat and empirical reward r_hat.
• Sampling error: the gap between the empirical Bellman operator t_hat and the true operator t, bounded using the same concentration inequalities from part 1 and scaling as 1/sqrt(n).
• Approximation error: the gap between the fitted q_hat_{k+1} and the exact backup t q_hat_k, assumed bounded by some epsilon_k in the infinity norm.
• Contraction property: the Bellman backup shrinks the infinity-norm distance between two Q-functions by a factor of gamma, which is what lets the per-iteration errors be summed into a geometric series.
• Forgetting the initialization: as the number of iterations grows, the effect of the starting q_hat_0 vanishes because gamma^k goes to zero.
• Combined error bound: the asymptotic error is proportional to 1/(1-gamma) times the largest per-iteration error, and because sampling error itself contains a 1/(1-gamma) factor, the overall bound is quadratic in the effective horizon 1/(1-gamma).

Before you watch

• Watch part 1 of this lecture first: it introduces the concentration inequalities and the 1/(1-gamma) horizon argument reused here.
• Review the Bellman contraction property and the definition of fitted Q-iteration from the earlier value-based methods lectures.

Check your understanding

1. What are the two distinct sources of error in idealized fitted Q-iteration, and which one comes from having a finite number of samples?
2. How does the contraction property of the Bellman backup let you turn a per-iteration error bound into a bound on the limiting error as k approaches infinity?
3. Why does the final error bound end up quadratic in 1/(1-gamma) rather than linear?
4. Why does the lecture call the infinity-norm assumption on fitting error a strong assumption, and what alternative does it mention?