Lecture 11, Part 5: Model-Based RL with Image Observations

What this lecture covers

Closing out Lecture 11, this part addresses model-based RL when observations are images rather than compact states, as in Atari or robot camera input. It follows the uncertainty-aware modeling discussion from earlier in the lecture and extends the model-based RL recipe to handle high-dimensional, partially observable, redundant image data.

After watching, you'll be able to explain why images complicate model-based RL, describe how latent state-space models split the problem into an observation model and a dynamics model, and contrast learning a compact latent space with predicting directly in image (observation) space.

Key ideas

• Why images are hard: high dimensionality, redundant pixel information, and typically partial observability (a single frame may not reveal velocity or direction) all complicate direct modeling.
• POMDP framing: with images, the underlying problem is a partially observed MDP; the true state s_t is unknown, and only observations o_t and actions a_t are available.
• Latent state-space models: learn an observation model p(o_t | s_t), a dynamics model p(s_t+1 | s_t, a_t), and typically a reward model p(r_t | s_t, a_t), separating the high-dimensional part (images) from the temporal part (dynamics).
• Approximate posterior (encoder): since true states are unobserved, training requires an expected log-likelihood over an approximate posterior q(s_t | ...), ranging from a simple single-step encoder to a more accurate but harder-to-train full smoothing posterior.
• Deterministic encoder simplification: for near-fully-observed problems, using a deterministic encoder s_t = g(o_t) removes the need for the expectation and lets the whole system train end-to-end with backpropagation.
• Direct image-space prediction: an alternative that skips learning a compact latent state altogether and predicts p(o_t+1 | o_t, a_t) directly, which can work well in visually complex scenes where a compact state space is hard to define.

Walkthrough

Why complex image observations are hard (0:00)

The lecture lists the specific challenges images introduce for model-based RL: high dimensionality, pixel redundancy, and partial observability. It introduces the POMDP graphical model and proposes splitting the problem into a low-dimensional dynamics model and a separate high-dimensional observation model that maps latent states to images.

State-space (latent space) models (2:26)

The lecture works through training a latent state-space model: an observation model, a dynamics model, and a reward model, all trained via an expected log-likelihood since the true states are unknown. It discusses the tradeoff between a full smoothing posterior (most accurate, hardest to train) and a simple single-step encoder (easiest to train, least accurate), noting more partially observed problems need more accurate posteriors.

Approximate posteriors and the deterministic special case (4:42)

For simplicity, the lecture focuses on a deterministic single-step encoder s_t = g(o_t), which removes the expectation from the training objective and reduces it to a differentiable sum of dynamics and reconstruction ("autoencoder") log-probabilities, trainable directly with backpropagation. This whole latent model plugs into the model-based RL version 1.5 (MPC) loop from earlier in the lecture, just with observations instead of states.

Examples: embed to control and follow-on work (12:15)

The lecture reviews the "Embed to Control" paper, which learns a stochastic latent space and plans through it with LQR, showing results on 2D navigation, an inverted pendulum, cart-pole balancing, and a simulated reaching task. It then describes a follow-on method that regularizes the latent space to be locally linear (suited to iterative LQR) and demonstrates it on real robotic manipulation tasks, including stacking Lego blocks and pushing objects to goals, trained efficiently from real-world images.

Direct prediction in image space (roughly 15:40)

As an alternative to latent state-space models, the lecture describes learning p(o_t+1 | o_t, a_t) directly in image space without an explicit compact state. This can work well in visually complex scenes with many objects where defining a compact latent space is difficult, illustrated with a robotic arm pushing objects and a reaching/grasping task guided by a target image.

Before you watch

• Watch Lecture 11, Parts 1 through 4 first for the model-based RL loop (version 1.5/MPC) and uncertainty-aware modeling that this part extends to image observations.
• Familiarity with autoencoders and the general idea of an encoder-decoder structure is helpful, though the lecture explains the needed pieces.

Check your understanding

1. What specific properties of image observations make direct model-based RL harder than with compact state vectors?
2. Why does a latent state-space model need both an observation model and a dynamics model, rather than one combined model?
3. What is the tradeoff between a full smoothing posterior and a single-step encoder when approximating the true state posterior?
4. How does using a deterministic encoder simplify training a latent state-space model?
5. Under what circumstances might predicting directly in image space work better than learning a compact latent state?