Lecture 15, Part 1: What Is Offline Reinforcement Learning?

What this lecture covers

This lecture opens a multi-part unit on offline reinforcement learning, motivating why learning from a fixed, previously collected data set (rather than through active interaction) is important for getting RL to generalize the way supervised deep learning does. It contrasts the closed-world settings where online RL succeeds, like AlphaGo, with the open-world generalization achieved by supervised methods trained on huge, diverse, but static data sets like ImageNet.

The lecture defines the offline RL problem formally, distinguishes it from off-policy evaluation, and argues with concrete robotic examples that offline RL can in principle do better than the best behavior in the data set through generalization and "stitching" together partial trajectories. It then demonstrates, using a large-scale robotic grasping study and a controlled half-cheetah experiment, that naively applying standard Q-learning offline causes catastrophic overestimation of Q-values, setting up the distributional shift problem that later lectures in the unit will address.

Key ideas

• Offline RL definition: given a fixed data set D of transitions collected by an unknown behavior policy pi_beta, learn the best possible policy without any further interaction with the environment.
• Behavior policy pi_beta: the policy (possibly a mix of humans, scripted controllers, or past RL runs) that collected the data set; unlike imitation learning, it is not assumed to be an expert.
• Off-policy evaluation versus offline RL: off-policy evaluation estimates the return of a given policy from D, while offline RL uses the data to learn an improved policy, often relying on some form of policy evaluation internally.
• Trajectory stitching: an offline RL algorithm can, in principle, combine partial trajectories (for example, one that goes from A to B and another from B to C) via dynamic programming to synthesize better behavior than any single trajectory in the data set.
• Distribution shift: standard Q-learning trains the Q-function under the data distribution pi_beta but queries it under the current policy pi_nu during bootstrapping, and because pi_nu is optimized to maximize the Q-function, it is effectively incentivized to find actions that fool the Q-function into outputting erroneously high values.
• Out-of-distribution actions: the central problem in offline RL is that the learned policy may select actions dissimilar to anything in the data set, for which the Q-function's estimates are unreliable and cannot be corrected because there is no way to collect more data to verify them.
• Empirical evidence of failure: a large-scale robotic grasping study and a controlled half-cheetah experiment both show that offline Q-learning produces wildly overestimated Q-values (for example, predicting a return of 10^7 while the actual return is negative) alongside poor actual performance.

Walkthrough

Why offline RL matters for generalization (0:01)

The lecture opens by contrasting the closed-world settings where deep RL succeeds (like AlphaGo, which never has to handle unexpected real-world disruptions) with the open-world generalization of supervised deep learning trained on huge internet-scale data sets. It argues that scaling online or off-policy RL to collect data sets of comparable size and diversity is impractical because it would require collecting an ImageNet-scale data set on every training run, motivating the idea of reusing large, previously collected data sets the way supervised learning does.

Defining offline RL and its variants (8:06)

The lecture formally defines the offline RL setting: a fixed data set D of state, action, next state, and reward transitions collected by an unknown behavior policy pi_beta, with the goal of learning the best possible policy without further interaction. It distinguishes offline RL from the related but easier problem of off-policy evaluation, and clarifies that "best possible policy" really means the best policy supported by the evidence in the data set, not necessarily the true optimal policy for the underlying MDP.

Why offline RL can beat the best behavior in the data (14:08)

The lecture argues that offline RL is not the same as imitation learning, and that a good offline RL algorithm can outperform the best individual trajectory in the data set through generalization and stitching: combining a driver who is good at merging with another who is good at intersections, or connecting separately observed sub-trajectories into a longer successful path through a maze. A robotic grasping example illustrates this concretely, where prior data of opening and closing drawers, unrelated to any single grasping demonstration, lets the policy learn to retrieve an object from a closed drawer it never saw solved end-to-end.

Distributional shift and the overestimation problem (21:15)

The lecture presents evidence that naive offline Q-learning goes badly wrong. In a large Google robotic grasping study, purely offline training reached an 87% success rate versus 96% with a small amount of additional online fine-tuning, more than tripling the failure rate. A controlled half-cheetah experiment shows the same effect more starkly: offline actor-critic training produces negative rewards that get worse over training, while the learned Q-values are absurdly inflated. The lecture explains this as a distribution shift problem: the Q-function is accurate under pi_beta but is queried under pi_nu, which is explicitly optimized to find actions that maximize (and thus can exploit inaccuracies in) the Q-function, similar to how adversarial examples are constructed for neural networks.

Before you watch

• This lecture assumes familiarity with Q-learning, actor-critic methods, and off-policy algorithms from earlier in the course.
• Reviewing the homework 3 Q-learning implementation is useful, since it is referenced directly as a point of comparison.
• No prior exposure to offline RL specifically is needed; this lecture is the introduction to the topic.

Check your understanding

1. Why can't classic on-policy or off-policy RL algorithms easily scale to ImageNet-sized data sets the way supervised learning can?
2. What is the difference between off-policy evaluation and offline reinforcement learning?
3. Explain the "stitching" intuition for how offline RL can produce behavior better than any single trajectory in the data set.
4. Why does standard Q-learning tend to badly overestimate Q-values when trained purely offline, and how does this connect to adversarial examples in supervised learning?