Lecture 1: LLM Reasoning

What this lecture covers

This is the opening lecture of Berkeley's LLM Agents course, taught by Dawn Song, with a guest talk by Denny Zhou of Google DeepMind on LLM reasoning. After a short course introduction covering the syllabus, guest speakers, and the key challenges in building LLM agents (reasoning and planning, learning from feedback, multimodal grounding, multi-agent collaboration, and safety), the lecture turns to Zhou's core topic: why language models need to generate intermediate reasoning steps to solve problems reliably, and how that idea evolved into techniques like chain-of-thought prompting and self-consistency.

By the end, you should understand why few-shot prompting without reasoning steps often fails on simple tasks, why adding a worked-out reasoning process fixes this, and why sampling multiple reasoning paths and taking the most common answer (self-consistency) improves accuracy further. You'll also see documented limitations: models get distracted by irrelevant context, generally cannot reliably self-correct without external feedback, and are sensitive to the order in which a problem's information is presented.

Key ideas

• Few-shot prompting without reasoning: giving a model input-output examples alone (no explanation of how to get the answer) mimics standard supervised learning and often produces wrong answers on tasks that require multiple steps.
• Chain-of-thought (intermediate steps): adding a demonstrated reasoning process before the answer in the prompt lets models solve tasks like letter concatenation or math word problems far more reliably, sometimes from a single example.
• Zero-shot prompting ("let's think step by step"): appending this phrase alone, without any worked examples, can trigger step-by-step reasoning in a model.
• Analogical reasoning: prompting a model to first recall a related problem or method before solving the actual problem outperforms plain zero-shot chain-of-thought.
• Self-consistency: sampling several independent reasoning paths for the same question and taking the most frequent final answer, rather than trusting a single generated path, substantially raises accuracy.
• Distraction by irrelevant context: adding unrelated sentences to a math problem measurably lowers model accuracy, similar to findings in human psychology studies.
• Limits of self-correction: without an external signal (like a unit test or an oracle) telling the model whether its answer is right, asking a model to "review and correct" its own answer can turn correct answers into incorrect ones.
• Order sensitivity: reordering the sentences of a math or logic problem, without changing its meaning, can drop model accuracy by 10 to 30 points, because models tend to process information sequentially rather than reason flexibly.

Walkthrough

Course introduction and why agents matter (0:00)

Dawn Song introduces the teaching staff and frames the course: large language models that only map text to text are limited, and the course's focus is LLM agents, systems that use an LLM as the reasoning and planning engine while interacting with external tools, environments, and other agents. She lists the key open challenges the course will address, including reasoning and planning quality, learning from environment feedback, multimodal grounding, multi-agent collaboration, and safety and privacy, then previews the syllabus of model capabilities, agent frameworks, application domains, and ethics.

From data-hungry learning to reasoning (9:02)

Denny Zhou opens his talk by describing his years-long search for what's "missing" in machine learning, given that data-efficient learning from a handful of examples never really panned out despite decades of research effort. His answer: reasoning, not statistics, is what lets humans learn from just a few examples. He illustrates this with a "last letter concatenation" toy problem that is trivial for people but historically required large labeled datasets for machine learning models to solve well.

Discovering chain-of-thought (13:06)

Using the last-letter problem, Zhou shows that plain few-shot prompting (examples with only the final answer) produces wrong output, but adding a short worked-through reasoning process to each example fixes it, even with just one demonstration. He traces this idea back to earlier papers, including a 2017 paper on solving math word problems with intermediate steps, the GSM8K dataset from 2021, and Google Brain's "show your work" paper, before naming the general prompting technique chain-of-thought. He stresses that what matters isn't whether the model is fine-tuned or just prompted, but whether intermediate steps are present at all.

Structured decomposition and generalization (23:20)

Zhou describes least-to-most prompting, which explicitly breaks a complex problem into simpler subproblems before solving them in sequence, inspired by George Pólya's classic book on problem solving. He shows this approach reaching near-perfect accuracy on compositional generalization tasks (like translating natural-language commands into action sequences) using a tiny fraction of the training data that specialized architectures needed.

Why intermediate steps work, and triggering reasoning without examples (30:33)

He summarizes theoretical work showing that transformers generating intermediate steps can solve any problem within a fixed computational class using constant depth, while transformers forced to output direct answers may need much greater depth or fail outright. He then covers "let's think step by step," a zero-shot prompt that triggers reasoning without any examples, and analogical reasoning prompting, where the model is asked to first recall a related problem before solving the target one, which outperforms plain zero-shot chain-of-thought.

Reasoning without any prompting (37:47)

Zhou presents chain-of-thought decoding, where reasoning emerges by examining alternative first tokens during greedy decoding rather than through any prompt. He shows that paths which happen to contain step-by-step reasoning are associated with higher model confidence in the final answer, suggesting pretrained models already "know" how to reason internally.

Self-consistency (43:57)

He reframes the decoding problem from a probability standpoint: a model should maximize the probability of a correct final answer, which requires summing over all reasoning paths that lead to it, not just picking the single most likely path. Since that sum is intractable to compute exactly, self-consistency approximates it by sampling many reasoning paths and taking the most frequent final answer. He extends this idea to free-form answers with "universal self-consistency."

Limitations: distraction, self-correction, and order sensitivity (51:13)

The final section covers three failure modes: models lose accuracy when irrelevant sentences are added to a problem, similar to human distractibility; models generally cannot reliably self-correct their own answers without an external oracle signal (such as a unit test), and asking them to "review and correct" can just as easily damage a correct answer as fix a wrong one; and reordering a problem's sentences or logical rules, without changing meaning, can cause large accuracy drops because models tend to process information in the order given rather than reorganizing it.

Before you watch

• No prior lectures in this course are required, since this is the first session; general familiarity with how transformer-based language models generate text is helpful.
• It helps to know what few-shot prompting means (showing a model example input-output pairs in the prompt before asking it to answer a new question).

Check your understanding

1. Why does adding intermediate reasoning steps to few-shot examples improve model accuracy compared to showing only final answers?
2. What is the difference between chain-of-thought prompting and self-consistency, and why does self-consistency require sampling multiple times?
3. Why might asking a model to review and correct its own answer sometimes make a correct answer wrong?
4. How does the theoretical result about transformer depth explain why direct-answer generation struggles on some problems that chain-of-thought can solve?
5. Give an example of how reordering the information in a problem, without changing its meaning, can hurt a model's accuracy.