Guest Lecture: Aviral Kumar on Offline RL for Pre-training

What this lecture covers

This CS285 guest lecture by Aviral Kumar (a major contributor to the course's offline RL and RL theory material) asks how offline reinforcement learning can support the pre-train-then-fine-tune paradigm that has driven progress in language and vision models, but for decision-making. Instead of collecting arbitrary data and directly producing a policy that maximizes reward, the goal is to pre-train generalist models on broad, loosely related data and then fine-tune them cheaply for a specific downstream task.

The talk is organized into three parts: scaling up offline RL to larger models, using arbitrary data sources (specifically human videos) to pre-train useful representations for robots, and building fine-tuning algorithms that avoid a performance collapse ("unlearning") when adapting a pre-trained offline policy with limited online interaction. Each part is grounded in concrete experiments, from multi-game Atari benchmarks to real robot manipulation tasks.

Key ideas

• Pre-train-then-fine-tune for decision-making: rather than training directly on task-specific data, train generalist policies, features, or representations on broad data, then fine-tune on a target task, mirroring the foundation-model recipe from NLP and vision.
• Offline RL scaling problem: unlike supervised imitation learning, which improves steadily with larger models on a multi-game Atari benchmark, off-the-shelf offline RL methods like conservative Q-learning (CQL) can get worse as model capacity increases.
• Implicit regularization mismatch: supervised learning's implicit regularizer (from stochastic gradient descent) tends to favor low-norm features, but Q-learning's bootstrapped targets introduce a second regularization term that pushes feature norms up, conflicting with the first and hurting scaling.
• Fixing the scaling gap: explicitly adding back a term that cancels this conflicting effect lets CQL scale with model size on multi-game Atari, becoming (per the lecture) the first Q-learning-based offline method to beat the behavior policy on that benchmark with a large model.
• Pre-training robot policies with human video: because video data of humans interacting with the world vastly outnumbers robot interaction data, the lecture proposes pre-training a visual encoder on video with goal-conditioned value functions, then using that encoder to initialize offline RL on robot data.
• Choosing the right value function family: modeling values only for optimal goal-reaching policies (rather than one fixed reward function, or all policies at once) strikes a practical balance between representation breadth and usefulness for downstream robot control.
• The fine-tuning "unlearning" problem: naively continuing offline RL training with newly collected online data can cause a sharp temporary performance drop, traced to CQL pushing Q-values to be uniformly small, which lets out-of-distribution actions appear erroneously good once real rewards are observed.
• Calibrated Q-values (Cal-QL) fix the dip: constraining learned Q-values to never fall below a reference Q-function, such as the behavior policy's return-to-go estimate, prevents the erroneous "high peak" problem and produces faster, monotonic improvement during online fine-tuning.

Walkthrough

Motivation: pre-training and fine-tuning for offline RL (0:00)

The lecture opens by contrasting the standard offline RL pipeline (data to policy) with the pre-train-then-fine-tune pipeline used in modern machine learning (broad data to generalist model to fine-tuned model). It sets up three questions to address: how to use arbitrary data sources, how to scale to larger models, and how to fine-tune pre-trained initializations effectively.

Scaling offline RL to larger models (5:34)

Using a multi-game Atari benchmark (roughly two billion transitions across games), the lecture shows that imitation-learning-style methods scale well with model size while off-the-shelf CQL degrades. It traces this to a difference in implicit regularization between supervised and Q-learning objectives: bootstrapped targets introduce a term that effectively increases feature norms, counteracting the norm-reducing regularization that helps supervised learning generalize.

Improving scaling with regularization (21:18)

Adding an explicit regularizer that cancels the problematic second term restores the expected benefit of scaling: performance increases with model capacity in a pattern similar to supervised learning, and the resulting large-model CQL variant surpasses the behavior policy's performance on the multi-game Atari benchmark, which prior offline RL methods had not achieved.

Pre-training with human video data (26:21)

The lecture pivots to using data beyond robot demonstrations, specifically internet-scale human video, since it vastly exceeds available robot interaction data despite lacking actions and differing in embodiment. A value function trained on video (accounting for the world's dynamical structure via Bellman backups) is proposed as a way to learn a useful visual encoder without needing actions.

Designing goal-conditioned value functions for video (32:23)

The lecture works through the trade-off in choosing what value function to train on video: a single fixed reward function is too narrow, modeling all policies is too broad and includes unrealistic random behavior, so the chosen approach models values for goal-reaching reward functions under optimal policies only, balancing breadth and usefulness.

Evaluating video pre-training on real robots (37:28)

The learned value functions show more consistent, monotonically increasing trends on held-out and out-of-distribution robot trajectories compared to alternatives, and policies fine-tuned from this video-pretrained encoder generalize better across object and gripper variation and distractor objects than policies trained without video, or with other video pre-training methods like autoencoders or contrastive learning.

The unlearning problem in online fine-tuning (41:30)

Naively continuing to run CQL or IQL with newly collected online data produces either a sharp performance dip followed by recovery (CQL) or slow, low-slope improvement (IQL). Tracing this to a sudden shift in average Q-value magnitude, the lecture explains that CQL's pessimistic regularizer finds Q-functions with the smallest possible values among those fitting the data, which makes previously unseen actions look erroneously attractive once real online rewards arrive.

Calibrated Q-learning (Cal-QL) for stable fine-tuning (49:37)

The fix constrains learned Q-values to stay above a reference function, such as the behavior policy's return-to-go estimate, preventing erroneous peaks from forming during online updates. This produces faster, more consistent improvement than CQL or IQL, the smallest cumulative regret among compared methods, and is demonstrated on a real robot microwave-opening task where the policy improves steadily from an offline initialization over about 20,000 online steps.

Before you watch

• Review the course's two lectures on offline RL, including conservative Q-learning, since this talk builds directly on that material without re-deriving it.
• Recall goal-conditioned RL and contextual policies from the transfer learning lecture, which underpin the video pre-training value function design.
• Basic familiarity with implicit regularization in supervised deep learning is helpful for the scaling analysis section.

Check your understanding

1. Why does off-the-shelf offline RL (CQL) scale worse with model size than supervised imitation learning on the multi-game Atari benchmark, according to the implicit regularization analysis?
2. Why is modeling value functions for a single fixed reward function, or for all possible policies, each a poor choice when pre-training visual representations on human video?
3. What causes the "unlearning" dip observed when fine-tuning CQL online, and how does constraining Q-values to a reference function address it?
4. Why might human video data be useful for pre-training robot policies even though it contains no actions and humans differ from robots in embodiment?