Lecture 22, Part 4: Gradient-Based Meta-Reinforcement Learning

What this lecture covers

This segment presents gradient-based meta-learning, best known as model-agnostic meta-learning (MAML), as an alternative to the RNN-based approach from the previous part. It reframes pre-training and fine-tuning as a form of meta-learning and asks whether the fine-tuning process itself can be optimized directly.

The core idea is to make F_theta, the function that adapts to a new MDP, literally be a gradient update: starting from initial parameters theta, one policy-gradient step on a task's objective produces the adapted parameters phi_i. Meta-training then searches for an initialization theta such that a single gradient step, for any task drawn from the distribution, leads to high reward. The lecture connects this to supervised MAML, discusses why it works despite looking deceptively simple, and shows an ant robot example adapting to run in different directions after one gradient step.

Key ideas

• Pre-training and fine-tuning as meta-learning: gradient-based meta-learning treats the fine-tuning step itself as the thing being optimized, rather than just the initial features.
• F_theta as a gradient step: the adaptation function is defined as theta + gradient of J_i(theta), i.e. one (or more) policy-gradient updates evaluated on the specific task's objective.
• Second-order optimization: meta-training searches for an initial theta such that applying a gradient step on any meta-training task increases that task's reward as much as possible, which is a form of second-order optimization.
• Same computation graph as an RNN meta-learner: gradient descent inside F_theta can be treated as just another differentiable architecture, implementable with automatic differentiation, though policy-gradient second derivatives require extra care.
• Favorable inductive bias: because the adaptation mechanism is a real gradient step, models trained this way can often take more gradient steps at test time than they were meta-trained with, unlike RNN-based meta-learners whose adaptation is a fixed forward pass.
• Multiple gradient steps are possible: the method is not limited to one gradient step, though the math for multiple steps and for policy gradients specifically is more involved.

Before you watch

• Watch Part 3 of this lecture, which introduces the general meta-RL framework and the RNN-based alternative that this segment contrasts with.
• Review the pre-training and fine-tuning discussion from Part 1, since this lecture directly builds on that idea.
• Familiarity with policy gradient methods is needed to follow how F_theta is defined as a gradient update.

Check your understanding

1. How does model-agnostic meta-learning (MAML) redefine F_theta compared to the RNN-based meta-learner from the previous part?
2. Why is meta-training in MAML described as a second-order optimization problem?
3. What practical advantage do gradient-based meta-learners have over RNN-based ones at meta-test time?
4. In the ant robot example, what happens to the meta-trained policy before any adaptation, and how does one gradient step change its behavior?