Lecture 14: Reasoning and Language Model Agents

What this lecture covers

This lecture, presented as an open research discussion by Shikhar Murty, tackles two applications of language models that go beyond producing plausible text: reasoning through multi-step problems, and taking actions in an environment as an agent. The first half surveys prompting techniques that elicit reasoning-like behavior (chain-of-thought, self-consistency, problem decomposition) and methods for training smaller models to reason, then critically examines whether that reasoning is genuine or memorized. The second half reframes decision-making in environments as a language modeling problem and surveys how language model agents are built, trained, and evaluated.

It follows the lecture on efficient training and builds on earlier lectures covering pretraining and instruction tuning; distillation methods here (Orca) echo the RLHF and fine-tuning ideas from those lectures. After watching, you should be able to name several prompting strategies for eliciting reasoning, describe at least one method for testing whether a model's stated reasoning is faithful to its actual answer, and explain how a language model can be turned into an agent that interacts with an environment like a web browser.

Key ideas

• Three kinds of reasoning: deductive (rules plus premise to a firm conclusion), inductive (observations to a likely generalization), and abductive (an observation to a plausible explanation); the lecture focuses on informal, multi-step deductive-style reasoning.
• Chain-of-thought prompting gets a model to produce intermediate reasoning steps before an answer, either via in-context examples or the simple instruction "let's think step by step."
• Self-consistency samples multiple reasoning paths and answers for the same question and takes the majority answer, which outperforms both standard chain-of-thought and simple ensembling over different prompts.
• Least-to-most prompting decomposes a question into sub-questions, answers each, and conditions the final answer on those sub-answers, sometimes generalizing to problems needing more reasoning steps than were shown in the prompt.
• Distillation and self-training: Orca fine-tunes a small model on detailed explanations generated by GPT-4, while ReST (reinforced self-training) has a model generate its own rationales, filters them by whether they reach the correct answer, fine-tunes on the filtered set, and repeats, sometimes outperforming human-written rationales.
• Faithfulness tests cast doubt on chain-of-thought: forcing a model to answer after only part of its rationale, or deliberately corrupting steps of the rationale, sometimes doesn't change the final answer, suggesting the stated reasoning can be a post-hoc explanation rather than the actual basis for the answer.
• Counterfactual evaluation exposes memorization: performance drops noticeably on tasks shifted out of the training distribution (base-9 arithmetic instead of base-10, an unfamiliar alphabet, a world where "corgis are reptiles"), even though humans show little such drop, suggesting some apparent reasoning ability reflects memorized patterns.
• Language model agents reframe decision-making as generating a trajectory of actions conditioned on a goal and past observations, allowing a model to act as a policy through prompting in a loop rather than through reinforcement learning or hand-built semantic parsers.
• Synthetic demonstration generation: since human demonstrations don't scale across many environments, one method has an agent explore randomly, uses a second model to guess what instruction a trajectory accomplished, and iteratively refines trajectory-instruction pairs to build training data.

Walkthrough

Kinds of reasoning and chain-of-thought prompting (1:07)

The lecture opens by distinguishing deductive, inductive, and abductive reasoning, and formal versus informal reasoning, before narrowing its focus to informal, multi-step deductive reasoning. It then asks whether large language models, already known to produce plausible text continuations, can reason. Chain-of-thought prompting is introduced as the most basic test: providing in-context examples with explicit reasoning steps, or simply prompting with "let's think step by step," elicits reasoning-like rationales before an answer.

Self-consistency and problem decomposition (5:12)

Self-consistency improves on chain-of-thought by sampling multiple rationale-and-answer pairs for the same question and selecting the most common answer by majority vote, which the lecture notes outperforms simple ensembling across differently prompted copies of the same model. Least-to-most prompting takes a different approach, decomposing a question into sub-questions, answering each in sequence, and building the final answer from those sub-answers; one experiment showed this decomposition strategy generalizing to problems requiring more reasoning steps than appeared in the prompt, though the lecture notes it's unclear whether this decomposition is fundamentally necessary once chain-of-thought prompts are engineered carefully.

Distilling reasoning into smaller models (10:16)

Rather than relying on very large models, Orca fine-tunes a smaller 13-billion-parameter Llama model on detailed step-by-step explanations generated by GPT-4 in response to instructions drawn from the Flan V2 collection. On the Big Bench Hard benchmark, which covers tasks like evaluating boolean expressions, date understanding, and interpreting SVG-described geometric shapes, Orca outperformed both ChatGPT and a comparable instruction-tuned model (Vicuna) that lacked this explanation-based fine-tuning. ReST (reinforced self-training) goes further by having a language model generate its own rationales, filtering them by whether they reach a correct answer (for example, on the GSM8K grade-school math dataset), fine-tuning on the filtered rationales, and repeating this cycle; after enough iterations, models trained this way sometimes outperformed models fine-tuned on human-written rationales.

Testing whether chain-of-thought reasoning is faithful (20:31)

The lecture then questions whether these rationales reflect the model's actual reasoning process. One experiment forces the model to answer after only part of its rationale ("early exit") and checks whether the answer changes; on several datasets it doesn't, suggesting the full rationale wasn't necessary to reach that answer. A second experiment deliberately corrupts steps of the rationale and checks whether the final answer changes; again, corrupting early steps sometimes has little effect on some datasets. Both results suggest that at least some chain-of-thought rationales function as post-hoc explanations rather than the actual computation behind the answer.

Counterfactual evaluation and memorization (25:34)

To separate genuine reasoning from memorized patterns, the lecture describes testing models on counterfactual versions of familiar tasks: arithmetic in base 9 instead of the much more common base 10, logic problems set in an invented world (where corgis are reptiles), and analogical reasoning tasks with an altered alphabet or altered rule. Performance drops noticeably on these out-of-distribution variants, while human performance on the same counterfactual tasks barely changes, suggesting that some of a model's apparent reasoning ability reflects patterns seen during training rather than a systematic, general reasoning capability.

From reasoning to language model agents (30:41)

The lecture's second half introduces agents: a model that receives observations and a language instruction from an environment and issues actions, such as clicking or typing in a browser. It reviews three pre-language-model approaches to instruction following: treating instructions as a machine translation problem into logical forms executable against a database, inferring an executable "plan" from instruction-action pairs and running it through a separate execution model, and using reinforcement learning to directly map instructions and observations to actions via a reward signal.

Modern language model agents and benchmarks (38:49)

The current approach treats decision-making as autoregressive generation: a model conditions on the instruction, the environment's current state, and the trajectory so far, and predicts the next action, factorizing the trajectory probability into environment dynamics and an agent policy. In its simplest form this means prompting a language model in a loop with an action space defined in text, essentially chain-of-thought prompting applied to acting rather than answering. The lecture surveys benchmarks of increasing realism: MiniWoB (simple, short-horizon simulated browser tasks), WebArena (closer approximations of real e-commerce and social media sites with multi-tab browsing), and WebLINX (real websites with multi-tab browsing and the ability for the agent to request information from a human). Even on the simplest of these, zero-shot performance from the best language models is far from perfect, and performance drops sharply as tasks require more sequential actions.

Training agents with synthetic demonstrations, and multimodal agents (46:55)

Since collecting human demonstrations doesn't scale across the many possible environments and tasks, one method lets an agent explore an environment randomly to generate trajectories, then uses a second language model to guess what instruction each trajectory accomplished, iteratively refining trajectory-instruction pairs (or discarding and relabeling failed attempts) until a filter judges them a good match. Fine-tuning or using these synthetic demonstrations as in-context examples improved performance on MiniWoB. The lecture closes by noting that feeding full HTML into a model's context doesn't scale, motivating vision-language approaches like LLaVA (which fine-tunes an image encoder and text decoder jointly on GPT-4-generated image descriptions and Q&A) and Pix2Struct (pretrained by masking parts of website screenshots and predicting the corresponding HTML), both later adapted for agents that act directly on screenshots. Despite this progress, even the best models still make basic errors, such as typing an email address into a password field and being unable to recover, and there remains a large gap between human and model task success rates on realistic benchmarks.

Before you watch

• Review the earlier lectures on pretraining and instruction tuning (RLHF), since the distillation methods here (Orca) build directly on those ideas.
• Familiarity with in-context learning and prompting basics will help with the chain-of-thought and self-consistency sections.
• No specific background in reinforcement learning is required, but knowing roughly what a reward signal is helps with the historical section on pre-language-model agents.

Check your understanding

1. What is the difference between deductive, inductive, and abductive reasoning, and which form does the lecture focus on?
2. How does self-consistency differ from simply ensembling multiple prompted copies of the same model, and why does the lecture say it performs better?
3. Describe one experiment the lecture uses to test whether a model's chain-of-thought rationale is faithful to how it actually reached its answer.
4. Why does testing a model on base-9 arithmetic reveal something that testing it on base-10 arithmetic cannot?
5. How does the synthetic demonstration generation method decide whether a randomly explored trajectory is a "good" example to train on, given that there is no ground-truth instruction to check against?