Lecture 1: Introduction, Part 1

What this lecture covers

This opening part of the first CS285 lecture asks why reinforcement learning (RL) is needed at all, using robotic grasping as a motivating example. It contrasts RL with the supervised, density-estimation approach behind today's image and language generation models, then traces RL's roots back to animal behavior research and control/optimization.

After watching, you should be able to explain why supervised learning struggles with tasks like robotic grasping, describe the difference between learning to imitate data and learning to maximize a reward, and name the two historical lineages that shaped modern deep RL.

Key ideas

• Grasping as a motivating problem: picking up objects has many special cases (rigid vs. deformable, center of mass) that are hard to hand-engineer, and even humans can't reliably label "correct" grasp locations, making standard supervised learning a poor fit.
• Reward instead of labels: in RL, robots collect their own trial data labeled only with outcomes (success/failure), and a reward function scores these outcomes instead of providing ground-truth answers.
• Generative models as density estimation: recent AI advances in image and text generation are framed as estimating P(X) or P(Y|X) from large datasets, learning to reproduce the distribution of human-generated content.
• Two lineages of RL: one traces to psychology and animal behavior studies (B.F. Skinner), the other to control, optimization and evolutionary algorithms (e.g., Karl Sims's simulated creatures).
• Deep RL as a combination: modern deep reinforcement learning merges classical RL's algorithmic ideas with large-scale optimization and neural network function approximation.
• Emergent behavior: results like AlphaGo's "move 37" are notable because they were not copied from human play; RL can discover solutions a person would not have chosen, which matters for building systems that respond intelligently to novel situations.

Before you watch

• No prior lecture is required; this is the first video of the course.
• Basic familiarity with supervised learning (inputs, outputs, training on labeled data) helps you follow the contrast being drawn.

Check your understanding

1. Why is robotic grasping difficult to frame as a standard supervised learning problem?
2. What does a reward function provide that a labeled dataset does not?
3. What are the two historical disciplines that influenced modern reinforcement learning?
4. Why does the lecture consider AlphaGo's "move 37" significant?