Lecture 2: LLM Agents History and Overview

What this lecture covers

Shunyu Yao, one of the researchers behind ReAct and WebShop, gives a history-driven overview of LLM agents. He first defines what "LLM agent" even means by separating text agents, LLM agents, and reasoning agents, then traces two older paradigms, rule-based text agents (like ELIZA) and reinforcement-learning text agents, before showing how question answering research with language models split into a "reasoning" thread (chain-of-thought) and an "acting" thread (tool use, retrieval), which converged into ReAct-style reasoning agents. The lecture then covers long-term memory, why reasoning agents differ fundamentally from symbolic and deep-RL agents, and closes with newer application domains and five open research directions.

This is a conceptual, historical companion to the previous lecture's technical focus on reasoning. After watching, you should be able to explain what distinguishes a reasoning agent from earlier agent paradigms, describe how ReAct interleaves thinking and acting, and name the open problems in training, interfaces, and benchmarking that current agent research is tackling.

Key ideas

• Text agent vs. LLM agent vs. reasoning agent: a text agent takes and produces language; an LLM agent is a text agent powered by an LLM; a reasoning agent is an LLM agent that also uses the LLM to reason, not just to act.
• Two older paradigms: rule-based chatbots like ELIZA and reinforcement-learning text agents both required domain-specific engineering and didn't generalize; LLMs promised generality through pretraining and prompting instead.
• Reasoning and acting as separate threads: question-answering research split into augmenting models with reasoning (chain-of-thought) and augmenting them with acting (retrieval-augmented generation, tool calls, program generation), each solving different weaknesses.
• ReAct: interleaving thought and action in the same generated trajectory lets reasoning adjust plans when actions return unexpected results, and lets actions supply real-world information that reasoning alone can't produce.
• Reasoning as an internal action: unlike a fixed action space (like left/right/up/down in a game), the "thinking" action available to a reasoning agent is language itself, an unbounded space that changes only the agent's own context, not the external world.
• Long-term memory: because a model's context window is append-only, limited in size, and doesn't persist across tasks, agents need a separate long-term memory (such as reflection on past failures) to actually learn from experience without retraining.
• Reflexion: an agent that fails a task can generate a written reflection on why it failed and store it, then use that memory to do better on a retry, without updating any model weights.
• Three components of any agent: memory (where information is stored), action space (what the agent can do), and decision-making (how it chooses actions) are sufficient to describe any agent, digital or physical.

Walkthrough

What is an LLM agent (0:00)

Yao opens by asking the class to define "agent," pointing out it is a notoriously broad term. He builds a definition layer by layer: an agent is an intelligent system interacting with an environment; a text agent is one where both observation and action are language; an LLM agent is a text agent that uses an LLM to act; a reasoning agent additionally uses the LLM to reason.

Text agents before LLMs (4:02)

He reviews ELIZA (1960s rule-based chatbot) and reinforcement-learning text agents, noting both required task-specific engineering, extensive training, or handcrafted reward signals, and neither generalized across domains the way LLMs, trained generally on next-token prediction, promised to.

Reasoning and acting as separate threads in QA research (8:07)

Using question answering as a simplifying case study, Yao shows how researchers separately tackled the weaknesses of LLMs answering questions: chain-of-thought reasoning for problems needing multiple steps, retrieval-augmented generation for outdated or missing knowledge, and tool calls or generated code for computation. He notes the field felt "scattered," with a different technique for each benchmark, motivating a search for a unifying abstraction.

ReAct: interleaving reasoning and acting (15:10)

He introduces ReAct, where a prompted trajectory alternates thoughts, actions, and observations. Using a worked example (whether $7 trillion could buy Apple, Nvidia, and Microsoft by market cap), he shows how reasoning adapts the plan when a search returns unhelpful or adversarial results, and how acting supplies real-time information reasoning alone lacks. He argues this pattern generalizes beyond question answering to any task that can be framed as a text game, including robotics and video games, once observations and actions are converted to text.

Why reasoning agents are structurally different (23:18)

Yao contrasts three agent paradigms: symbolic AI agents that map observations to fixed logical rules, deep reinforcement-learning agents that map observations to a fixed-size embedding, and reasoning agents that use language, an open-ended representation, as the intermediate step between observation and action. He argues this is why reasoning agents transfer across domains far more easily than hand-engineered or narrowly trained agents.

Long-term memory (26:24)

He explains that a language model's context window functions as short-term memory: append-only, size-limited, and non-persistent across tasks. He introduces reflection (used in the Reflexion method) as a simple form of long-term memory, where an agent writes down why it failed a coding task and reuses that note on a retry, and briefly describes more elaborate memory systems such as Voyager's library of learned skills in Minecraft and Generative Agents' episodic and semantic memory for simulated town residents.

New applications: digital automation and beyond (43:42)

Yao surveys tasks reasoning agents have opened up beyond QA and games: WebShop (agents browsing a shopping site), software engineering benchmarks like resolving GitHub issues, and scientific discovery agents that propose and test new chemical compounds through lab feedback. He contrasts these practical, open-ended tasks with older, narrow, synthetic agent benchmarks.

Five open directions for future research (54:47)

He closes with five research directions: training models specifically for agent behavior rather than only prompting general-purpose models; designing better interfaces between agents and environments (illustrated by a file-search interface tuned for agents rather than humans); and robustness, illustrated by tau-bench, a customer-service benchmark where what matters is solving a task reliably every time, not just once out of many samples, unlike typical "pass@k" coding benchmarks. He notes that today's models show a concerning trend of declining reliability as they're sampled more, the opposite of what real-world deployment needs.

Before you watch

• Watching Lecture 1 (LLM Reasoning) first is useful, since this lecture assumes familiarity with chain-of-thought reasoning as a foundation for the "reasoning" half of ReAct.
• No coding background is required, but familiarity with the basic idea of reinforcement learning (actions, environments, rewards) helps with the historical comparisons.

Check your understanding

1. What distinguishes a "reasoning agent" from a plain "LLM agent" in Yao's three-level definition?
2. Why did rule-based and reinforcement-learning text agents fail to generalize across domains before LLMs?
3. How does ReAct's interleaving of thought and action let an agent recover from a bad or adversarial search result?
4. Why is reasoning described as an "internal action" that only changes the agent's own context rather than the external environment?
5. Why does tau-bench measure whether an agent solves a task every time it's sampled, rather than the "pass@k" metric common in coding benchmarks?