Lecture 22, Part 1: Transfer Learning and Domain Adaptation

What this lecture covers

This lecture opens Berkeley CS285's unit on transfer and meta-learning, asking how an RL agent can reuse experience from source tasks to learn new target tasks faster. It reframes earlier exploration examples, like Montezuma's Revenge, as transfer problems: humans solve the game quickly because they bring prior knowledge about keys, ladders and skulls, and the lecture asks whether RL algorithms can acquire and reuse similar knowledge.

After defining source and target domains and terms like zero-shot, one-shot and few-shot transfer, the lecture works through practical techniques: pre-training and fine-tuning, handling domain shift with invariance assumptions and adversarial discriminators, adjusting rewards when dynamics differ between domains, and using domain randomization to make policies robust. It closes with a short introduction to multitask transfer and contextual and goal-conditioned policies, setting up the following lecture on meta-learning.

Key ideas

• Source and target domain: the source domain (or domains) supplies the experience you train on; the target domain is the new task you want to do well on.
• Shot terminology: zero-shot means the policy works on the target task with no additional training; one-shot, few-shot and many-shot describe how many attempts on the target task are allowed before good performance.
• Domain shift: a representation learned in the source domain, especially from images, often does not transfer cleanly because the target domain looks or behaves differently.
• Invariance assumption: transfer works well if you can find a featurization Z = f(X) where the distribution of Z is the same across domains and Z still predicts the label or Q-value, meaning everything that differs between domains is irrelevant.
• Domain-adversarial training: a discriminator tries to tell source from target activations while the encoder is trained (via a reversed gradient) to fool it, producing domain-invariant features.
• Reward shaping for dynamics differences: when the domains differ physically rather than just visually, you can penalize the agent for taking actions in the source domain that would be impossible in the target domain, approximated with two discriminators.
• Domain randomization: training across a wide range of simulated physical or visual parameters produces policies that generalize better to the target domain, sometimes even to parameters that were never randomized.
• Contextual and goal-conditioned policies: adding a context variable (one-hot task id, image, text, or goal state) to the policy's input turns multitask learning into a single joint MDP where the context is sampled at the start of each episode.

Walkthrough

Motivation: why transfer matters (0:00)

The lecture revisits Montezuma's Revenge to show why some games are hard for RL despite looking simple to humans: the reward signal gives little guidance, and success depends on prior knowledge about game conventions like keys, doors and skulls. This motivates treating the problem as transfer learning, where an agent reuses knowledge from other tasks instead of exploring from scratch.

Defining transfer learning and shot terminology (5:02)

Transfer learning is defined as using experience from one or more source MDPs to learn faster or perform better on a target MDP. The lecture introduces the zero-shot, one-shot, few-shot and many-shot vocabulary for describing how much interaction with the target domain is allowed, and distinguishes transfer learning's target-only focus from continual or lifelong learning, which also cares about retaining source-domain performance.

Forward transfer, multitask transfer and meta-learning overview (8:06)

Three broad strategies are laid out: forward transfer (train on a source task, then run or fine-tune on the target task), multitask transfer (train on many source tasks so the target task falls within their range), and meta-learning (train specifically to make future adaptation fast). The lecture notes that transfer learning outside meta-learning tends to be ad hoc and domain-specific, while meta-learning offers more general principles, which is why later lectures spend more time there.

Pre-training, fine-tuning and domain shift (12:09)

Borrowing the standard supervised-learning workflow of pre-training a representation and fine-tuning a few layers, the lecture explains why this is harder in RL: domain shift between source and target observations, and the tendency of optimal policies in fully observed MDPs to become deterministic, which kills exploration during fine-tuning.

Invariance and domain-adversarial methods (15:11)

The invariance assumption formalizes when transfer should work: if a featurizer Z = f(X) makes the distribution of Z match across domains while preserving everything needed to predict the output, transfer can succeed. This motivates domain-adversarial training, where a discriminator that tries to classify source versus target activations is opposed by the feature encoder via gradient reversal, producing features that hide which domain they came from. The lecture warns this can misfire if target-domain data quality (e.g. bad demonstrations) leaks into the invariance objective.

Handling dynamics differences and randomization (23:15)

When source and target domains differ in dynamics, not just appearance, forcing pure invariance can hide functionally important differences. Instead, the lecture describes adding a reward penalty, estimated via two discriminators, for actions that would be impossible in the target domain. It then turns to domain randomization: training across a wide distribution of physical or visual parameters (such as varying mass, friction or lighting) produces policies robust to the real target domain, and randomizing several parameters can create robustness even to parameters that were left fixed.

Multitask transfer and contextual policies (36:19)

The closing section reframes multitask RL as a single joint MDP where the initial state distribution samples which task to attempt, and introduces contextual policies that take a task descriptor (one-hot vector, image, or text) as extra input. It highlights goal-conditioned policies as a convenient special case, where the context is a goal state and the reward depends on reaching it, enabling zero-shot transfer to new goals but requiring extra tricks to train well.

Before you watch

• Review the exploration lectures' discussion of Montezuma's Revenge, since it is used again here as the running motivating example.
• Be comfortable with the standard actor-critic and Q-learning setup, since contextual policies simply extend the state input.
• Familiarity with basic adversarial training (as in GANs) helps with the domain-adversarial discriminator section.

Check your understanding

1. What does the invariance assumption require of a featurization Z = f(X), and why can it fail when source and target domains differ in dynamics rather than just appearance?
2. Why does fine-tuning a deterministic optimal policy on a new target domain often struggle to explore?
3. How does domain randomization make a policy robust to physical parameters that were never explicitly randomized?
4. What distinguishes a contextual policy from a goal-conditioned policy, and why can goal conditioning enable zero-shot transfer?