Lecture 16, Part 4: Offline RL in Practice, Applications, and Open Problems

What this lecture covers

This closing part of the offline RL unit steps back from algorithmic derivations to practical guidance: which offline RL algorithm to reach for given your constraints, illustrated as a simple decision tree comparing conservative Q-learning, implicit Q-learning, advantage-weighted actor-critic, and model-based options. It also explains why offline RL is attractive for real-world applications where online exploration is costly or dangerous, contrasting a slow, throwaway online RL workflow with a much more reusable offline workflow.

The lecture then walks through several real robotics case studies, including large-scale grasping and off-road navigation projects, where reusing existing offline data let researchers test new hypotheses without new data collection. It closes with a discussion of open problems in the field: validation without online deployment, weak statistical guarantees, and the challenge of scaling offline RL to more domains.

Key ideas

• Algorithm selection rule of thumb: for pure offline training, conservative Q-learning (CQL) is simple and well-validated; implicit Q-learning (IQL) is more flexible; for offline training followed by online fine-tuning, advantage-weighted actor-critic (AWAC) or IQL tend to work better than CQL, which is often too conservative to fine-tune well; model-based methods like COMBO or the trajectory transformer are worth considering when a good dynamics model is feasible for the domain.
• Offline versus online real-world RL workflow: online RL in the real world requires safety instrumentation, autonomous resets, hand-designed reward functions, and slow iteration where each change requires rerunning the entire process from scratch; offline RL allows algorithm changes without recollecting data, and lets data sets be aggregated and reused across projects.
• Reward-free goal-conditioned relabeling: an offline data set collected for one purpose can be reused to train goal-conditioned policies without a hand-designed reward function, simply by rewarding similarity to a goal image, letting new hypotheses be tested without new data collection.
• Real robotics case studies: examples include a large multi-task grasping and rearrangement project where 12 tasks were relearned from existing data using only goal images, and a sequence of off-road navigation projects (delivery, goal search) that all reused a single 40-hour navigation data set collected years earlier.
• Open problems: the field still lacks an offline equivalent of a supervised learning validation set, meaning policy quality typically still requires costly or risky online evaluation; theoretical bounds on distributional shift remain loose; and offline RL has not yet been applied broadly outside a fairly narrow set of domains like robotics.

Walkthrough

Choosing an offline RL algorithm (0:01)

The lecture presents a practical decision tree for algorithm choice as of late 2021: CQL for pure offline training due to its simplicity and wide validation; IQL as a more flexible alternative that also works well with online fine-tuning; AWAC for training that will be followed by online fine-tuning (noting CQL tends to be too conservative for this); and model-based methods like COMBO or the trajectory transformer when a good domain model is achievable, with the caveat that this recommendation may change as the field evolves quickly.

Why offline RL matters for real-world deployment (3:03)

The lecture contrasts the slow, effortful loop required for real-world online RL, which needs safety mechanisms, autonomous reset procedures, hand-engineered rewards, and a full restart for every algorithmic change, with the offline RL workflow, where a growing data set can be reused indefinitely, reward changes don't require recollection, and only occasional online evaluation is needed to check progress.

Real robotics case studies (5:05)

The lecture describes a large Google project that reused an existing 12-task grasping and rearrangement data set to test whether goal-conditioned RL (rewarding similarity to a goal image rather than task-specific rewards) could learn the same tasks without any hand-designed reward function, and shows the resulting robot successfully grasping objects and rearranging them, such as moving a carrot onto a plate, based purely on a goal image. It also describes a chain of off-road navigation projects at Berkeley that reused a single 40-hour data set collected in 2020 to successively build a mail/pizza delivery system and a goal-search system, without collecting any additional real-world data.

Open problems and conclusions (9:07)

The lecture closes by naming several open problems: the lack of an offline analogue to a supervised learning validation set, meaning most offline RL evaluation still requires deploying the policy online; weak and incomplete statistical guarantees around distributional shift; and the fact that despite offline RL's broad potential applicability (medical diagnosis, logistics, algorithmic trading), it has so far mostly been demonstrated in robotics, leaving open how well current methods generalize to other domains.

Before you watch

• Watch Lecture 16, Parts 1 through 3 first, since this closing part assumes familiarity with CQL, IQL, AWAC, and the model-based methods it recommends in its decision tree.
• Familiarity with the offline RL motivation from Lecture 15, Part 1 is useful for understanding why the real-world workflow comparison matters.

Check your understanding

1. According to the lecture's rule of thumb, why might CQL be a poor choice if you plan to fine-tune your policy online afterward?
2. How does the offline RL workflow described in this lecture reduce the cost of iterating on an algorithm compared to real-world online RL?
3. How did the Google grasping project reuse existing data to test a hypothesis about goal-conditioned RL without collecting new data?
4. What is the main open problem the lecture identifies regarding how offline RL policies are currently evaluated, and why is it a problem?