Lecture 15, Part 3: Classic Offline RL with Linear Value Functions

What this lecture covers

This part of the lecture covers the second class of classic batch RL methods: those based on linear fitted value functions. Although modern offline RL uses deep neural networks rather than linear function approximation, the lecture argues these older methods are still worth understanding because their closed-form, least-squares solutions provide useful analysis tools and intuition for more modern algorithms.

Starting from feature matrices and linear models of reward and transitions, the lecture derives the least-squares temporal difference (LSTD) formula for value functions, extends it to Q-functions to get least squares policy iteration (LSPI), and shows how to make all of this fully sample-based. It closes by explaining why these linear methods, despite their elegance, do not solve the distributional shift problem that plagues offline RL, setting up the need for the more modern techniques covered in the next lecture.

Key ideas

• Feature matrix: a matrix Phi of size (number of states) by (number of features) that gives a compact feature vector for every state, used as the basis for all the linear approximations in this lecture.
• Linear reward and transition models: the reward vector and next-step features are each approximated as a linear function of Phi (via weight vectors w_r and a feature-space transition matrix P_phi), both solvable in closed form using ordinary least-squares normal equations.
• Vector Bellman equation: the value function for a fixed policy satisfies V = R + gamma * P * V, a linear system in V that can be solved directly by matrix inversion since I - gamma*P is always invertible.
• Least squares temporal difference (LSTD): substituting the linear reward and transition solutions into the feature-space Bellman equation yields a closed-form formula for the value function weights w_v, expressed purely in terms of features, without needing to separately estimate the reward or transition model.
• Sample-based ("empirical MDP") version: replacing per-state rows of the feature matrix with per-sample rows recovers the same LSTD formula using only a data set of transitions, at the cost of introducing sampling error.
• LSTD-Q and LSPI: because LSTD for value functions requires samples from the target policy's transition distribution (which offline data doesn't have), the lecture uses state-action features instead of state features, yielding LSTD-Q, and alternates evaluating and greedily improving the policy to get least squares policy iteration (LSPI), a genuine offline RL algorithm.
• Why linear methods don't fix distributional shift: because LSPI still takes an argmax over Q-values at each iteration, it suffers from the same action distributional shift problem as deep methods; it just doesn't show it as severely because linear function approximators generalize less aggressively than neural networks.

Walkthrough

Warm-up: linear models of reward and transitions (2:04)

The lecture introduces the feature matrix Phi and shows how to fit a linear reward model w_r and a linear feature-space transition matrix P_phi using ordinary least-squares normal equations, treating this purely as a modeling warm-up before connecting it back to value functions.

Deriving LSTD (9:07)

The lecture writes the value function Bellman equation in vector form, V = R + gamma * P * V, and shows it can be solved directly as a linear system since I - gamma*P is always invertible. Substituting in the least-squares solutions for the reward and transition models from the warm-up produces the least squares temporal difference (LSTD) formula, a closed-form expression for the value function weights that turns out to be model-free in effect, even though it was derived by way of a model-based argument.

Making it sample-based and building LSPI (13:12)

The lecture converts LSTD into a fully sample-based method by replacing per-state feature rows with per-sample rows from an offline data set of transitions, calling this the "empirical MDP." It then explains why value-function LSTD alone cannot evaluate policies other than the one that collected the data, motivating a switch to state-action features and Q-functions, which yields LSTD-Q. Alternating between computing w_q for the current policy and updating the policy to be greedy with respect to it gives least squares policy iteration (LSPI), a complete, if classic, offline RL algorithm.

Why linear methods still hit distributional shift (20:18)

The lecture closes by pointing out that despite all this derivation, LSPI still performs an argmax over Q-values during policy improvement, which is exactly the step where distributional shift causes trouble, since the algorithm is essentially performing empirical risk minimization and then querying the fit adversarially. It notes these linear techniques are valuable for analysis and for understanding where modern deep offline RL methods come from, but they are not, by themselves, adequate solutions to the distributional shift problem in practice.

Before you watch

• Review the earlier value-based and actor-critic lectures, since this lecture reuses the Bellman equation and Q-function machinery from those in vector and matrix form.
• Watch Lecture 15, Parts 1 and 2 first for the definitions of offline RL and the distributional shift problem this lecture revisits.
• Basic familiarity with least-squares regression and matrix inversion is assumed.

Check your understanding

1. How does the vector form of the Bellman equation allow the value function to be solved directly as a linear system?
2. What does the LSTD formula represent, and why is it described as effectively model-free even though it is derived from a model-based argument?
3. Why does LSTD for value functions fail to evaluate a policy other than the one that collected the data, and how does switching to Q-functions (LSTD-Q) fix this?
4. Why do linear methods like LSPI still suffer from distributional shift, even though they don't show the same dramatic overestimation seen with deep neural networks?