Lecture 1: Introduction, Part 2

What this lecture covers

This part covers the CS285 course structure (model-free RL, model-based RL, exploration, offline RL, inverse RL, meta-learning) and grading, then gets into the substance: what reinforcement learning is and how it differs from supervised learning. It closes with a tour of applications, from robotic manipulation to language model fine-tuning.

After watching, you should be able to state the definition of a policy and its role, contrast the i.i.d. assumption of supervised learning with the sequential, reward-only feedback of RL, and describe the credit assignment problem.

Key ideas

• Course structure: units move from foundations to model-free algorithms (Q-learning, policy gradient, actor-critic), model-based methods, then advanced topics (exploration, offline RL, inverse RL, meta-learning). Grading is 50% homework, 40% project, 10% quizzes.
• Reinforcement learning is two things: a mathematical formalism for learning-based decision making, and an approach for learning decision making from experience; the lecture stresses not conflating the problem with any particular solution method.
• Supervised learning's assumptions: it treats data as independent and identically distributed (i.i.d.) and assumes every input has a known correct label — assumptions that don't hold for interactive decision-making tasks.
• The RL loop: an agent observes a state s_t, chooses an action a_t, and the environment returns a new state s_{t+1} and a reward; the goal is a policy Pi_theta that maps states to actions and maximizes cumulative reward, not just immediate reward.
• Credit assignment: when an outcome is delayed, it's often unclear which earlier action caused a success or failure, unlike labeled supervised data.
• Reward framing across domains: dog training, robot locomotion, and inventory management are all reframed as (actions, observations, reward) tuples to show how broadly the RL formalism applies.
• Applications shown: robotic hammering and jumping, Atari play discovering the "tunnel through the bricks" strategy, warehouse robots sorting trash, RL-controlled traffic flow reducing jams, and RL fine-tuning of language models (RLHF) and image generation models.

Walkthrough

Course Overview (0:00)

The instructor lists the course units and grading breakdown, describes the five homeworks (imitation learning, policy gradients/Q-learning, actor-critic, model-based RL, offline RL), and explains the research-level final project, including proposal and milestone checkpoints meant to give students feedback before the final report.

Reinforcement Learning (3:10)

The lecture defines reinforcement learning as both a formalism and a class of solution methods, then walks through the RL loop: an agent picks actions, the environment returns states and rewards, and the objective is a policy that maximizes cumulative reward over time rather than reward at a single step.

Supervised Learning (7:17)

By contrast, supervised learning is defined through its i.i.d. and fully-labeled-data assumptions. The lecture explains why these assumptions break down for sequential decision problems: past actions affect future inputs, and only outcomes (not correct actions) are observed, creating the credit assignment problem.

Examples (9:18)

The lecture works through examples — training a dog, controlling a robot, managing warehouse inventory — mapping each to actions, observations, and rewards, then shows applied results: robotic hammering and quadruped jumping, Atari game play, robotic trash sorting, RL-based traffic regulation, and RL used to fine-tune language models and image generation models toward human-preferred outputs.

Before you watch

• Watch Part 1 of Lecture 1 first for the motivating grasping example and historical background.
• Basic familiarity with the idea of training a model from a labeled dataset will help the supervised-vs-RL comparison land.

Check your understanding

1. Why can't reinforcement learning rely on the i.i.d. assumption that supervised learning makes?
2. What is the credit assignment problem, and why does it make RL harder than standard classification?
3. In the traffic example, what did the RL-controlled car optimize for instead of its own speed?
4. How was reinforcement learning used to improve the image generation example with the dolphin and bicycle?
5. What are the five homework topics listed for this course?