Lecture 1: Introduction, Part 3

What this lecture covers

This closing part of Lecture 1 argues for why deep reinforcement learning is worth studying now. It builds on Richard Sutton's "Bitter Lesson" essay, reframes all of machine learning as decision making, and surveys open problems around reward design, imitation, prediction, and using pretrained models for robot control.

After watching, you should be able to explain Sutton's distinction between "learning" and "search," describe why reward functions are often hard to define, and list several supervision signals beyond raw reward (demonstrations, prediction, transfer, pretrained models) that reinforcement learning can draw on.

Key ideas

• The Bitter Lesson: Richard Sutton argues the two approaches that scale indefinitely are learning and search; deep RL combines learning (extracting patterns from data) with search (using computation/optimization to reach better decisions), not just learning plus more compute.
• ML as decision making: reframing perception systems as decision-making systems (a car-detector's output ultimately drives some downstream action) suggests most machine learning problems are reinforcement learning problems where supervised labels are a convenient shortcut.
• Reward is often hard to define: examples like pouring water or a cheetah hunting a gazelle show that useful reward signals are frequently sparse or delayed, so humans and animals likely rely on additional signals, not trial-and-error from reward alone.
• Alternative supervision sources: imitation learning, inverse reinforcement learning (inferring reward from demonstrated behavior), learning to predict future observations, unsupervised feature learning, transfer learning, and meta-learning all supplement reward maximization.
• Predictive models enable emergent tool use: a robot trained to predict future images from actions can plan novel behavior, such as picking up an improvised tool to move objects it couldn't reach directly.
• Leveraging pretrained models: systems like RT-2 combine a language model and vision-language model pretrained on internet data with robot action fine-tuning, letting robots generalize to instructions (like "move the banana to the different object") never seen in robot-specific training data.
• The "one learning algorithm" hypothesis: evidence such as ferrets regaining vision through the auditory cortex, and tongue-based visual perception, suggests the brain may use a general-purpose learning mechanism adaptable to different sensory inputs, which the lecture connects to the case for deep RL as general-purpose building blocks.

Walkthrough

Reinforcement learning can discover new solutions (0:00)

The lecture contrasts data-driven density estimation (which reproduces human-like outputs) with RL's ability to discover better-than-human solutions, then introduces Sutton's Bitter Lesson and clarifies that "search" means optimization/computation, not literal tree search.

How do we build intelligent machines? (9:45)

Drawing on a quote from neuroscientist Daniel Wolpert, the lecture proposes that machine learning exists to produce decisions, not just predictions, and that essentially all ML tasks can be viewed as decision-making problems in disguise.

Where do rewards come from? / Imitation learning (12:15)

The lecture discusses the difficulty of specifying reward functions, sparse and delayed rewards in human and animal behavior, and alternative supervision sources: imitation learning (including inferring intent rather than copying actions literally) and inverse reinforcement learning applied to robots.

Prediction for real-world control (17:21)

The lecture covers predictive models that forecast future sensory input from actions, showing older robotic manipulation results and more recent diffusion-based video prediction models producing realistic driving and robotic footage.

Leveraging advances in pretrained models (19:19)

The RT-2 model is presented as an example of combining internet-pretrained language and vision-language models with robot action fine-tuning, enabling generalization to novel instructions and reasoning tasks.

Before you watch

• Watch Parts 1 and 2 of Lecture 1 first; this part builds directly on the supervised-vs-RL distinction and course motivation introduced there.

Check your understanding

1. In Sutton's Bitter Lesson, what does "search" mean, and how is it different from just adding more data or compute?
2. Why might reward-only trial and error be an implausible model for how a cheetah learns to hunt?
3. What is the difference between imitation learning that copies actions and imitation that infers intent?
4. How does the RT-2 example show a robot generalizing beyond its own training data?