Lecture 17, Part 1: RL Theory

What this lecture covers

This is the theory lecture of CS285. Rather than presenting a new algorithm, it asks what kinds of guarantees reinforcement learning methods can actually offer, and why those guarantees always rest on strong, somewhat unrealistic assumptions. The lecture works through one concrete case in detail: a tabular, model-based method that estimates transition probabilities by counting, and then measures how far the resulting Q-function is from the true one.

After watching, you should be able to state the difference between sample-complexity, regret, and policy-suboptimality questions, explain why exploration is usually studied separately from learning, and follow the chain of concentration inequalities and linear-algebra manipulations that produces a bound on q_pi - q_hat_pi in terms of the number of samples and the effective horizon 1/(1-gamma).

Key ideas

• Sample complexity: asks how many samples an algorithm needs so that its estimate is within epsilon of the true value with probability at least 1 - delta.
• Regret: measures how much reward is lost while running an exploration algorithm, as opposed to how accurate the final estimate is.
• Oracle/generative model assumption: lets the analysis sample any state-action pair as many times as needed, removing exploration difficulty so sample complexity of learning can be studied on its own.
• Concentration inequality: a bound (such as Hoeffding's inequality) on how fast a sample average concentrates around the true mean as the number of samples grows.
• Simulation lemma: relates the true Q-function and the Q-function of a learned model through the difference between the true and learned transition matrices, weighted by the true value function.
• Effective horizon: 1/(1-gamma) behaves like the horizon of a discounted infinite-horizon problem, and terms of this form recur throughout the bounds.
• Qualitative, not guaranteed, conclusions: theory in RL gives rough guidance about how error scales with problem parameters, not proof that an algorithm will work.

Walkthrough

Questions RL theory asks (0:23)

The lecture opens with a menu of theoretical questions: how close a learned Q-function is to the optimal one after a given number of samples, how the policy induced by that Q-function compares to the optimal policy (a regret-style question), and how the regret of an exploration method grows with time. It notes that upper-confidence-bound exploration methods typically have regret that grows logarithmically in the time horizon t. The lecture states it will mostly focus on sample-complexity questions.

Assumptions needed for analysis (3:49)

Full deep RL cannot be analyzed in the most general setting, so any useful theory needs strong assumptions. Because worst-case exploration is extremely pessimistic, exploration is usually analyzed separately from learning. One common simplification is the generative model (oracle exploration) assumption: the algorithm can sample the next state for any state-action pair as often as it likes, which sidesteps exploration difficulty entirely so the remaining question is sampling error.

What theoretical results are actually for (6:48)

The lecture is explicit that no current deep RL method has a real guarantee of working every time, and that claims of "provably good" RL algorithms should not be taken at face value: the assumptions behind them are always unrealistic. What theory does provide is qualitative guidance, such as whether error grows or shrinks with a larger state space, more iterations, or a larger discount factor. This is compared to idealized-engine analysis in physics, which describes limits and trends rather than predicting a real engine's exact efficiency.

Setting up the sample-complexity analysis (10:18)

The lecture defines a simple tabular, model-based algorithm: estimate the transition probabilities p_hat(s'|s,a) purely by counting transitions, then compute q_hat_pi exactly under that estimated model. The goal is to isolate how sampling error in p_hat propagates into Q-function error, borrowing tools from supervised learning theory. Hoeffding's inequality is introduced first, for estimating the mean of a bounded random variable, showing that the number of samples needed for a given error scales roughly as 1/epsilon^2. A related concentration inequality for categorical variables (used to bound the error of p_hat in total variation distance) is derived next, giving an analogous bound in terms of the number of states.

The lecture then expresses the Bellman equation in vector/matrix form, q_pi = r + gamma * P_pi * q_pi, which can be solved as q_pi = (I - gamma * P_pi)^-1 * r, and the same relation holds for the learned model with p_hat in place of p. Two lemmas are proved: the simulation lemma, which expresses q_pi - q_hat_pi in terms of (p - p_hat) weighted by the true value function v_pi, and a bound showing that applying (I - gamma * P_pi)^-1 to a vector can amplify its infinity norm by at most a factor of 1/(1-gamma), connecting this factor to the effective horizon of a discounted problem.

The simulation lemma proof (31:31)

The lecture walks through the algebraic derivation of the simulation lemma step by step: substituting the matrix expressions for q_pi and q_hat_pi, inserting an identity term, and collecting terms until the difference q_pi - q_hat_pi is expressed as gamma * (I - gamma * P_hat_pi)^-1 * (P_pi - P_hat_pi) * q_pi, which simplifies to a form involving v_pi.

Combining the pieces into a final bound (46:20)

Putting the simulation lemma and the norm-amplification lemma together, plus the concentration inequality on p_hat, yields a bound showing q_pi - q_hat_pi is bounded by a term proportional to gamma / (1-gamma)^2 times the total variation error in p_hat. The key qualitative conclusions: error shrinks at the same 1/sqrt(n) rate seen in supervised learning, but it grows quadratically in the effective horizon 1/(1-gamma), meaning each Bellman backup compounds error over the horizon. The lecture closes by extending the result from q_pi - q_hat_pi to bound q_star - q_hat_star (using a supremum inequality) and finally to bound the suboptimality of the actual policy obtained under the learned model, showing it is at most twice the original epsilon bound.

Before you watch

• Be comfortable with the Bellman equation and Q-value iteration from earlier lectures in this course.
• Some familiarity with basic probability (expectation, concentration inequalities) will make the derivations easier to follow.
• Review the matrix/vector notation for MDPs (P, V, Q as vectors and matrices), since the whole derivation is done symbolically.

Check your understanding

1. Why do researchers typically separate the analysis of exploration from the analysis of learning, and what does the generative model assumption let them avoid?
2. What is the practical difference between asking "how close is q_hat_pi to q_pi" and asking "how close is the true value of the learned policy to q_star"?
3. In the final bound on q_pi - q_hat_pi, why does the error grow quadratically rather than linearly in the effective horizon 1/(1-gamma)?
4. Why does the lecture caution against calling any deep RL algorithm "provably good"?