Guest Lecture: Dorsa Sadigh on Interactive Learning

What this lecture covers

This guest lecture closes out the course by asking how robots can learn what people actually want, beyond standard demonstrations or hand-written rewards. Dorsa Sadigh, a Stanford researcher working on human-robot interaction, walks through her lab's work on interactive learning: eliciting reward functions from pairwise human preferences, actively choosing which questions to ask, and, in the second half, using large language models and vision-language models as stand-ins for human feedback.

The talk assumes familiarity with reinforcement learning and reward functions from earlier in the course. After watching, you should understand why demonstration data is often insufficient on its own, how preference queries can be turned into an active-learning problem over reward weights, and two different ways researchers are currently trying to bring large pretrained models into robot learning.

Key ideas

• Interactive learning: learning a policy, reward function, or representation from data that a human actively or passively provides, rather than from a fixed dataset alone.
• Reward function from pairwise comparisons: showing a person two trajectories and asking which they prefer, then updating a hypothesis over reward weights based on the response.
• Active query synthesis: instead of picking from a fixed library, generate the next pair of trajectories that maximizes the expected volume removed from the space of possible reward functions.
• Boltzmann-rational human model: treats human preference responses as noisy rather than perfectly consistent, so each answer reweights rather than eliminates hypotheses.
• LLM as reward proxy: prompting a language model to judge whether a rollout exhibits a property (for example, fairness in a negotiation) and using that judgment as a training signal for reinforcement learning.
• Grounding problem: LLMs and VLMs perform well on text-based or simulated tasks but are less reliable once a policy has to interact with real physical state.
• Two takes on foundation models for robotics: building a robotics-specific foundation model trained on large offline data, versus using existing LLMs and VLMs directly for tasks like reward design, planning, or common-sense reasoning.
• Pattern machines: LLMs can continue numeric sequences (positions, control trajectories, reward-weighted trajectories) even when the tokens carry no semantic meaning, suggesting a use beyond language understanding.

Walkthrough

Why demonstrations alone are hard to use (1:01)

Sadigh opens with an assistive-feeding robot that has to bring food to a person's mouth safely. Reinforcement learning is impractical here because there is no good simulator and real-world exploration risks hurting the person. Imitation learning is also harder than it sounds: collecting teleoperated data requires specialized hardware, and human demonstrations are "suboptimal" in ways that are hard to clean up, since people pause and think about other things while demonstrating. This motivates looking beyond demonstrations to other sources of human signal, including pairwise preference comparisons and physical corrections.

Learning reward functions from pairwise comparisons (7:02)

The lecture formalizes preference learning: a reward function is modeled as a linear combination of features, weighted by a vector W. Each preference query (comparing two trajectories) defines a hyperplane through the space of possible W vectors, and the person's answer indicates which side the true W likely lies on. Because human responses are noisy, the model reweights rather than deletes hypotheses on the wrong side. This reframes preference elicitation as an active-learning problem similar to recommendation systems, but applied to robot behavior instead of movies.

Actively generating informative queries (12:05)

Because robotics involves a continuous trajectory space rather than a fixed library of options, the lab's approach synthesizes new trajectory pairs rather than selecting among existing ones. The objective maximizes the minimum expected volume removed from the hypothesis space of W, subject to the trajectories satisfying robot dynamics constraints. Applied to a driving simulator, around 70 binary preference questions were enough to learn collision-avoidance driving behavior without any demonstrations. Sadigh argues active learning matters more in robotics than in typical machine learning settings, because the cost of an extra query is real time spent with a person and a physical system, not just compute.

Using an LLM as a reward proxy in negotiation (24:10)

The lecture shifts to a text-based negotiation game where two agents split shared items. Training an agent with plain reinforcement learning produces aggressive behavior because properties like politeness or fairness are never specified in the reward. Instead of asking a human to judge each rollout, the lab prompts a large language model with a description of the negotiation and a property (such as versatility) and uses the model's judgment as a training signal, effectively reward shaping. This LLM-in-the-loop training is described as the inverse of RLHF: rather than fine-tuning the LLM from human feedback, the LLM supplies feedback to train a separate reinforcement learning agent. Across several properties, this signal tracked ground-truth reward reasonably well and outperformed a supervised-learning baseline.

LLMs and VLMs as reward designers for robots (29:13)

Extending the idea to robotics, related work (in collaboration with Google) has language models output the weights of a reward function directly from high-level instructions such as "make the robot face the sunset," which is then optimized with model predictive control. Demonstrated behaviors include sitting like a dog and performing a moonwalk. Sadigh is careful to flag the grounding problem: these examples largely work in simulation or with simple, non-contact interactions, and using vision-language models as success detectors for tasks that require real state estimation remains unreliable, though improving.

Learning visual representations for robotics: Voltron (38:19)

Turning to pretraining, the lecture contrasts masked autoencoders (which preserve fine spatial detail but lose semantics) with contrastive models like CLIP (which capture semantics but lose spatial detail). Voltron, a representation-learning model trained on human videos, combines a masked-autoencoding backbone with language captioning and generation and multi-frame conditioning, aiming to capture syntax, semantics, and some dynamics together. Fine-tuned on a small number of demonstrations, it outperformed R3M and masked visual pretraining baselines on imitation-learning tasks, and showed zero-shot intent inference on both human and unseen robot videos.

Two takes on foundation models and LLMs as pattern machines (49:23)

Sadigh closes by framing two research directions: building a robotics-specific foundation model from large offline datasets (citing cross-embodiment efforts like RT-X and in-the-wild data collection like R2D2), versus directly repurposing existing LLMs and VLMs for tasks such as task planning, code generation, common-sense reasoning about a scene, and semantic manipulation. A final, more surprising finding is that LLMs can act as "pattern machines," continuing numeric sequences such as end-effector trajectories or reward-weighted rollouts even when the tokens have no semantic meaning, hinting at uses beyond language understanding.

Before you watch

• Be comfortable with reward functions, policies, and the basics of reinforcement learning from earlier lectures in this course.
• Some familiarity with RLHF (reinforcement learning from human feedback) helps, since the lecture repeatedly contrasts its methods with that framework.
• No prior exposure to LLMs or vision-language models is assumed; the lecture explains their role as it goes.

Check your understanding

1. Why does the lecture argue that active query generation matters more in robotics than in many other machine learning settings?
2. How does the Boltzmann-rational human model change how a preference response updates the hypothesis space for W?
3. In what sense is training an RL agent with LLM-judged rollouts described as "the opposite of RLHF"?
4. What is the grounding problem, and why does it limit using LLMs or VLMs as reward functions for real robots?
5. What weakness in masked autoencoders and contrastive models like CLIP does Voltron try to address, and how?