Lecture 23, Part 2: Three Perspectives on What RL Is

What this lecture covers

In the closing part of the final CS285 lecture, Sergey Levine steps back from technical challenges to discuss what reinforcement learning fundamentally is, offering three perspectives that shape which RL methods matter most for a given goal. This is less a technical walkthrough than a reflective closing argument, drawing on research examples and broader AI philosophy.

The first perspective treats RL as an engineering tool for inverting physics: given a simulator, RL replaces the hand-derived control laws of classical control theory. The second treats RL as the natural framework for learning in the messy real world, developed through Moravec's paradox and examples of robots learning from failure. The third, the lecture's most speculative, argues RL could become the universal mechanism underlying all machine learning, using large low-quality datasets to learn what is possible before specializing to what is wanted, illustrated with pre-training and language-model examples.

Key ideas

• RL as an inversion engine: given a simulator (a stand-in for equations of motion), RL replaces the manual calculus of deriving a control law, making it a powerful optimization tool for systems that can be characterized and simulated.
• Moravec's paradox: abstract reasoning tasks like chess or Go are comparatively easy for AI, while motor control and perception, which biology has spent millions of years optimizing, remain hard; the lecture illustrates this with the observation that a human still moves the chess pieces even when a computer picks the moves.
• Easy vs hard universes: "easy" problems like chess or rocket flight have no perception or motor-control challenge; "hard" problems, the messy, unpredictable real world, are where RL's ability to learn from experience should matter most.
• Multitask learning from failure: structuring a robot's tasks so that a failure in one task (like dropping a cup) becomes the starting state for another (picking the cup back up) enables long, unsupervised practice without a human resetting the environment.
• Behavioral priors for exploration: pre-training on many prior tasks can give a robot a prior over useful actions, so exploration in a new task looks purposeful rather than random from the start.
• RL as a universal decision-making layer: instead of treating large scraped datasets as something to imitate (as in next-token prediction), the argument is to treat them as evidence of what is possible, then use RL-like processes to select what is optimal for a given goal.
• Learning from preferences: pairwise human comparisons of trajectories (RLHF-style) offer a practical way to specify objectives that are hard to write as a closed-form reward, illustrated with a backflip example and with a language-model teaching agent that learns to ask clarifying questions.
• Yann LeCun's cake argument (presented critically): a rough bits-of-supervision argument suggesting self-supervised learning provides far more signal per data point than RL's sparse rewards, raising the open question of what role RL should play relative to self-supervised and imitation learning.

Walkthrough

RL as an engineering tool (0:00)

The lecture frames RL as another way to invert physics: where classical control derives a feedback controller from hand-written equations of motion, RL can derive a control law directly from a simulator, illustrated with rocket flight and quadrupedal locomotion examples. Its main limitation is that someone still has to characterize and build the simulator.

Moravec's paradox and real-world learning (6:02)

Using the Deep Blue and AlphaGo milestones (where a human still physically moved the pieces), the lecture introduces Moravec's paradox: abstract thought is comparatively easy for AI, while perception and motor control are hard, because evolution has spent far longer optimizing the latter. This motivates viewing RL as suited to "hard universes" full of unexpected situations, illustrated with the Robinson Crusoe analogy and an oil-tanker example.

Learning fully autonomously in the real world (15:11)

The lecture surveys open real-world learning questions: how to specify objectives without a score, how to learn continually without episodic resets, and how to remain robust to change. It presents research examples, including learning from human preferences (Christiano et al.), a multitask robotic hand setup where failures at one task create starting states for another, enabling roughly 60 hours of autonomous practice, and behavioral priors built from prior tasks that make exploration in new tasks look purposeful rather than random.

RL as a universal learning framework (22:15)

The lecture's most speculative section argues that RL could generalize the pre-training and fine-tuning recipe behind modern deep learning: instead of modeling a data distribution (as language models do with next-token prediction), an RL-like process could treat large low-quality datasets as evidence of what actions are possible, then use a modest amount of supervision to select what is optimal. Examples include goal-conditioned offline RL pre-training for robots and a language-model-based teaching agent trained with RL to ask short, targeted clarifying questions instead of producing long generic responses.

Closing reflections on the big picture (32:19)

The lecture closes by revisiting the course's opening question of whether learning is the basis of intelligence, discussing Yann LeCun's "cake" argument about bits of supervision per data point, and weighing self-supervised learning, imitation, and RL as candidate cores for general-purpose AI. It ends with research advice: pick problems with high upside, reduce big questions to falsifiable and measurable ones, stay open to changing the problem statement, take real-world applications seriously, and think big while starting with concrete, actionable goals.

Before you watch

• Watch Part 1 of this lecture (the challenges and open problems survey), since this part builds directly on its discussion of generalization and real-world learning.
• Familiarity with the exploration lectures and with RLHF-style preference learning helps with the examples in this part.

Check your understanding

1. According to Moravec's paradox, why might abstract reasoning tasks like chess be easier for AI than everyday physical tasks like walking or grasping?
2. How does structuring tasks so that failures become new starting states help a robot learn autonomously without human intervention?
3. In the "RL as universal learning framework" perspective, what is the proposed difference between using scraped data to model a distribution versus using it as evidence of what is possible?
4. What is Yann LeCun's bits-of-supervision argument, and what question does it raise about RL's role relative to self-supervised learning?