Lecture 21: RL with Sequence Models & Language Models, Part 1

What this lecture covers

This lecture opens a block on reinforcement learning with sequence models by asking what happens once an agent only sees observations instead of true Markov states. It works through why partially observed MDPs (POMDPs) behave in surprising ways, such as needing information-gathering actions and stochastic optimal policies, then asks which of the standard RL algorithm families still work if you naively substitute an observation for a state. It sits at the start of a sequence that later connects this material to training language models with RL.

After watching, you can explain why the Markov property matters for value-based and model-based RL but not for policy gradients, and describe two practical ways, learned latent state spaces and history states, to make RL work correctly under partial observability.

Key ideas

• Markov states vs. observations: a true state makes the future independent of the past, so past states add no information; observations generally lack this property, so past observations can carry extra information about the present.
• Information-gathering actions: under partial observability, an optimal policy can take actions that do not directly increase reward but reduce uncertainty about the state, something that never happens in fully observed MDPs.
• Stochastic optimal policies: POMDPs can require randomized behavior to be optimal, illustrated by a three-state MDP where no deterministic memoryless policy beats a 50/50 policy, whereas fully observed MDPs always admit a deterministic optimal policy.
• "Handling" partial observability: the target standard used throughout the lecture is recovering the best policy within the class of memoryless (reactive) policies, not necessarily the best policy overall.
• Policy gradients tolerate observations: the policy gradient derivation never relied on the Markov property, so it's valid to use observations directly, as long as the advantage estimator avoids a value function that assumes Markovian states.
• Value-based and naive model-based RL fail: Q-learning and one-step predictive models both assume that visiting the same state always has the same expected outcome regardless of history, an assumption observations violate, illustrated with a stuck-door example where a naive Markovian model wrongly predicts a locked door can eventually "unlock."
• History states restore correctness: because a sequence of observations up to time t contains the sequence up to t-1, treating the full observation history as the state satisfies the Markov property and lets all three algorithm families work correctly.

Walkthrough

From Markov states to partial observability (0:00)

The lecture reintroduces the distinction between fully observed and partially observed MDPs from earlier in the course. A true state makes previous states uninformative once you know the current one, but observations, such as an image of a cheetah chasing a gazelle with something briefly blocking the view, don't guarantee this, so a history of past observations can carry information the current one lacks. Examples span Atari games (mildly partial observability, often safe to ignore), driving with blind spots, first-person video games, interacting with other agents whose mental state is hidden, and dialogue or tool-use settings where conversation history matters.

Two properties unique to POMDPs (4:08)

Two behaviors that never appear in fully observed MDPs are introduced. Information-gathering actions, illustrated by a maze-solving agent that benefits from peeking over the maze walls even though that action doesn't move it toward the exit, can be optimal purely because they reduce uncertainty. Stochastic optimal policies, illustrated with a simple three-state MDP where the reward is always in the middle state and the agent gets no informative observation, show that a 50/50 randomized policy can strictly outperform any deterministic memoryless policy.

Testing policy gradients, value-based, and model-based methods (7:05)

The lecture asks, for each of the three main RL algorithm families, whether naively substituting an observation for a state still recovers the best memoryless policy. Policy gradients turn out to be safe because their derivation never used the Markov property; the plain Monte Carlo return and the causality trick both remain valid, but a value-function-based advantage estimator is not valid, since the value function conditioned only on the observation can be wrong when the true value depends on earlier observations. The same reasoning shows this baseline use is still unbiased, just not necessarily variance-optimal.

Why value-based and naive model-based RL fail (15:08)

Q-learning is shown to be invalid under partial observability because it assumes the value of an observation-action pair doesn't depend on how the agent got there, which is false whenever the observation is non-Markovian; a further argument notes that Q-learning always yields a deterministic policy, yet POMDPs can require stochastic optimal policies, so no amount of tuning can fix this gap. A locked-door example demonstrates the same failure for naive model-based RL: a Markovian model of "probability of passing through a door" cannot represent the fact that once a specific door is found locked, it stays locked, because the model conditions only on the current observation and action, not on prior attempts.

Fixing partial observability with state space models and history states (20:09)

Two remedies are presented. Learning a latent Markovian state space, using a sequence VAE similar to the ones from the variational inference lecture, lets a learned latent z_t stand in for the true state in any downstream algorithm, but can require solving a harder generative modeling problem than the RL task itself. The simpler alternative is to use the full observation history itself as the state: since the history up to time t contains the history up to t-1, it provably satisfies the Markov property. This is implemented by feeding a sequence model, RNN, LSTM, or transformer, into the Q function, policy, or dynamics model. The lecture closes on a practical note: naively storing full histories in a replay buffer costs quadratic memory in the horizon, but caching RNN or LSTM hidden states avoids this, a trick detailed in the paper "Recurrent Experience Replay in Distributed Reinforcement Learning."

Before you watch

• Review the definitions of MDPs and POMDPs, and the concept of the Markov property, from earlier in the course.
• Recall the policy gradient derivation and the role of baselines and the causality trick from the policy gradients lectures.
• Familiarity with sequence VAEs from the variational inference lecture will help with the state space model section.

Check your understanding

1. Why can an information-gathering action be optimal under partial observability but never under full observability?
2. Which part of the policy gradient estimator remains valid under partial observability, and which part does not?
3. Why does naive Q-learning with observations in place of states fail to recover the optimal policy in the locked-door example?
4. Why does using the full observation history as the state satisfy the Markov property, and what practical problem does this create for algorithms like DQN?