Lecture 22, Part 3: Meta Reinforcement Learning with RNNs

What this lecture covers

This segment carries the meta-learning framework from supervised learning into reinforcement learning. It defines meta-RL by analogy: where regular RL maximizes reward under a policy for a single MDP, meta-RL trains a function that reads an MDP's experience and produces policy parameters that achieve high reward on that MDP, assuming meta-training and meta-test MDPs are drawn from the same distribution.

The core of the lecture is a concrete instantiation: an RNN that reads a sequence of transitions (state, action, reward, next state) across episodes within an MDP, builds a hidden state summarizing what it has learned, and feeds that hidden state to a policy head. Using a mouse-and-cheese example, the lecture shows how this history-conditioned policy naturally learns to explore, because maximizing total reward over a concatenated "meta-episode" forces the policy to seek information early and exploit it later.

Key ideas

• Meta-RL as a learning function: meta-RL trains F_theta, which takes an MDP (via the agent's experience in it) and produces policy parameters phi_i that perform well on that MDP.
• Same-distribution assumption: meta-training MDPs and the meta-test MDP must come from the same distribution P(M), just as training and test sets must match in supervised learning.
• Context as inferred experience: a meta-RL policy is like a contextual policy, but instead of being told the context (as in multitask learning), the context is inferred from experience gathered in the current MDP.
• RNN encoder architecture: a recurrent network reads the sequence of transitions across episodes into a hidden state; a policy head then maps the current state plus that hidden state to an action.
• Meta-episodes and exploration: because the RNN's hidden state persists across episode boundaries, optimizing total reward over a concatenated meta-episode automatically teaches the policy to explore early episodes to gather information and exploit that information in later ones.
• No separate exploration mechanism needed: exploration emerges from standard RL training on the higher-level meta-episode problem, without any extra exploration bonus.

Walkthrough

From supervised meta-learning to meta-RL (0:01)

The lecture restates the generic meta-learning recipe and transposes it to RL: F_theta now takes an MDP and produces parameters phi_i for a policy that should achieve high reward on that MDP. Meta-training MDPs and the meta-test MDP are assumed to share a distribution, illustrated with examples like a robot performing different tasks or a half-cheetah running at different speeds.

Meta-RL as history-conditioned contextual policies (2:02)

Meta-RL is connected to contextual policies: F_theta effectively summarizes all experience gathered in the current MDP into a context that conditions the policy, analogous to how a contextual policy is conditioned on a task descriptor. The key difference from multitask learning is that the context is inferred from experience rather than given directly.

Instantiating the RNN encoder (5:04)

The lecture builds a concrete architecture: an RNN reads a sequence of transitions across episodes, producing a hidden state, which a policy head combines with the current state to choose an action. The parameters theta are the RNN and policy head weights; the adapted parameters phi_i are simply the RNN's hidden state plus the policy head's fixed weights.

The mouse-and-cheese example and emergent exploration (7:07)

A worked example shows a mouse seeking cheese whose location changes between MDPs. Because the RNN's hidden state is not reset between episodes, it can use the outcome of an earlier episode (for example, discovering the cheese by going up and right) to act correctly in later episodes within the same meta-episode, illustrating how the policy both explores and exploits.

Why this trains exploration automatically (9:09)

The lecture explains that concatenating episodes into a meta-episode turns exploration into an ordinary RL optimization problem: maximizing total reward over the meta-episode implicitly rewards early actions that gather useful information. This principle underlies various meta-RL architectures, including actor-critic and policy-gradient implementations and, more recently, Transformer-based encoders.

Before you watch

• Watch Part 2 of this lecture, which introduces the general meta-learning framework and RNN-based supervised meta-learning that this segment extends to RL.
• Review contextual and goal-conditioned policies from Part 1, since meta-RL is framed as a special case where the context is inferred rather than given.

Check your understanding

1. How does the definition of F_theta change when moving from supervised meta-learning to meta-RL?
2. Why does an RNN-based meta-RL policy need to read transitions across multiple episodes rather than resetting at each episode boundary?
3. In the mouse-and-cheese example, what allows the policy to behave correctly on a second episode after failing to find the cheese on the first?
4. Why does optimizing reward over a meta-episode automatically produce exploratory behavior, without an explicit exploration bonus?