Lecture 22, Part 2: What Is Meta-Learning?

What this lecture covers

This short segment opens the meta-learning portion of Berkeley CS285's transfer learning unit. It frames meta-learning as a logical extension of multitask learning: instead of just solving many tasks, the goal is to use those tasks to learn how to learn new tasks more quickly.

The lecture first motivates why meta-learning matters for deep RL, given how sample-hungry model-free methods are, then works through a concrete supervised-learning example (few-shot image classification) to demystify what "learning to learn" actually means mechanically, before RL versions are introduced in later parts.

Key ideas

• Meta-learning as learning to learn: after training on many tasks, the aim is to generalize the learning process itself, not just the solutions to those tasks.
• Why it helps RL: a meta-learned RL method can explore more intelligently, avoid actions known to be useless, and acquire useful features faster than learning a new task from scratch.
• Meta-training vs meta-testing: meta-training uses a set of training/test set pairs across many tasks (the source domains); meta-testing applies the learned procedure to a genuinely new task (the target domain).
• Meta-learning as a function: standard supervised learning maps an input x to a prediction y; meta-learning maps an entire training set plus a test input to a prediction, f(D_train, x_test) = y_test.
• RNN-based meta-learner example: a recurrent network reads in a sequence of training examples, produces a hidden state summarizing the task, and a small classifier uses that hidden state plus a test input to predict the label.
• Two levels of optimization: generic learning finds parameters that minimize training loss; meta-learning trains a function f_theta so that the parameters it produces perform well on held-out test data across many tasks.

Before you watch

• Watch Part 1 of this lecture first, since it defines source/target domains and transfer terminology that this segment builds on.
• Basic familiarity with recurrent neural networks is useful for following the RNN meta-learner example.

Check your understanding

1. How does meta-learning differ from ordinary multitask learning?
2. In the RNN meta-learner example, what do the terms theta and phi each represent?
3. Why might a meta-learned RL policy explore more efficiently than one trained without meta-learning?