Lecture 3: Agentic AI Frameworks - AutoGen and Multimodal Assistants

What this lecture covers

This session has two guest talks. Chi Wang (AutoGen-AI, formerly Microsoft Research) explains why he expects future AI applications to be agentic, then walks through AutoGen, a framework for building agents as a generic "conversable agent" abstraction and composing them through conversation patterns like two-agent chats, nested chats, and group chats. Jerry Liu (LlamaIndex) then covers how to move beyond basic retrieval-augmented generation (RAG) to build a multimodal knowledge assistant: parsing complex documents (tables, charts, images), indexing them for retrieval, and adding agentic reasoning on top so a system can plan, use tools, and handle harder questions than simple lookups.

After watching, you should be able to describe AutoGen's core programming model (define agents, then define how they talk), explain why multi-agent decomposition can outperform a single agent on complex tasks, identify the four main limitations of a naive RAG pipeline, and describe what a multimodal, agentic RAG pipeline adds on top of basic RAG.

Key ideas

• Agentic AI as a software architecture: beyond enabling natural-language interfaces and reducing human supervision, structuring an application as multiple communicating agents is itself a useful way to build software, since agents can decompose and recursively execute complex tasks.
• Conversable agent: AutoGen's core abstraction treats a language model, a tool, or a human as an interchangeable "agent" backend, so the same conversation patterns work regardless of what powers an agent.
• Conversation patterns: two-agent chats (e.g., a writer and a critic iterating), nested chats (a single agent internally runs a sequence of sub-agents, like SEO, legal, and ethics reviewers, before responding), and group chats (a manager dynamically selects which agent speaks next) are the building blocks for more complex workflows.
• Task decomposition improves reliability: splitting a task across specialized agents (for example, a code-writing agent and a separate safety-checking agent) measurably improved success rates over a single agent doing everything, especially for weaker models.
• Basic RAG's limitations: naive retrieval-augmented generation does primitive chunking that ignores tables and images, uses the model only for synthesis rather than reasoning, and is stateless across interactions, which limits it to simple, one-shot questions.
• Multimodal RAG: a good document parser extracts text, tables, and images as distinct elements; each element is indexed (directly or via generated text summaries), so retrieval can return both text and images to a multimodal model.
• Agentic RAG: adding a reasoning layer, such as a ReAct-style loop, query decomposition, or a router, on top of retrieval lets a system handle summarization, multi-document comparison, and multi-part questions that fixed-chunk retrieval alone cannot answer well.
• Constrained vs. unconstrained agent flows: a hand-coded router-plus-reflection pipeline is more reliable but less flexible than a general agent loop (like ReAct) that decides its own tool calls, which is more expressive but riskier and more expensive.

Walkthrough

Why agentic AI, and a live demo (0:00)

Chi Wang frames two motivating questions: what future AI applications look like, and how to let every developer build them. He argues generative AI's quality jump around 2022 makes agentic applications (personal assistants, web agents, software agents) newly viable, and shows a demo of a multi-agent system building a website from a natural-language request, including a self-healing step where the agents detect and repair a deliberately broken file.

AutoGen's multi-agent design and why decomposition helps (7:12)

Using a supply-chain optimization example (commander, writer, and safeguard agents), Wang shows how a user's natural-language question triggers a chain of sub-conversations: the writer proposes code, the safeguard checks it's safe, and the commander executes it and relays results back. He presents experimental results showing a multi-agent split between a writer and a safeguard agent produced meaningfully higher safety-check accuracy than a single agent handling both tasks, especially for weaker models like GPT-3.5.

Core AutoGen concepts and conversation patterns (21:24)

Wang reduces AutoGen programming to two steps: define agents, then define how they talk. He introduces the "conversable agent" abstraction (which can be backed by a model, a tool, or a human) and demonstrates conversation patterns: a two-agent writer/critic loop for reflection, a nested chat where a single agent internally consults multiple specialized reviewers, a chessboard tool-agent that keeps two language-model players from making illegal moves, and group chats where a manager dynamically picks the next speaker, optionally with constraints on allowed transitions.

Real-world AutoGen applications and open research questions (28:32)

Wang surveys community use cases: scientific discovery teams (protein and material design), a web agent (Agent E) achieving state-of-the-art results on a browsing benchmark using only HTML content, and construction-industry applications. He closes with ongoing AutoGen research, including AutoBuild and AdaptiveBuild, which automatically propose or dynamically assemble a team of agents for a given task rather than requiring a developer to hand-pick them.

Limits of basic RAG and multimodal document parsing (37:38)

Jerry Liu introduces LlamaIndex and the goal of building a knowledge assistant over large, messy document collections. He walks through the "basic RAG" pipeline (parse, chunk, embed, vector search) and lists its four limitations: primitive data processing that ignores tables and images, using the LLM only for synthesis rather than reasoning or planning, and being stateless across interactions. He argues document parsing quality is foundational, since a poor parser feeds hallucinated or malformed text into an LLM regardless of the model's quality.

Building multimodal RAG (47:43)

Liu describes indexing distinct representations, direct embeddings for text chunks, generated text summaries linking back to tables and images, since these "nodes" are what a vector database actually stores. At retrieval time, both text and image content tied to a matched node can be passed to a multimodal model (he notes most current frontier models accept both text and images), producing a pipeline that can answer questions grounded in diagrams and tables, not just prose.

Agentic RAG: adding reasoning on top of retrieval (52:46)

Liu explains that naive RAG fails on summarization, comparison, and multi-part questions because it only retrieves a fixed set of chunks without reasoning about the question first. He contrasts constrained flows (a hand-written router plus a validation step, more reliable but less flexible) with general agent architectures like ReAct or LLMCompiler (more expressive, since the agent plans its own tool calls, but less reliable and more expensive). He describes LlamaIndex's event-driven workflow system as infrastructure for building either kind of flow and deploying it as a production service.

Applications and running agents in production (59:51)

Liu closes with example architectures, a researcher/writer/reviewer pipeline for report generation, and enterprise customer support, and describes deploying multi-agent systems as microservices behind APIs coordinated through a central message queue, including human-in-the-loop patterns where an agent pauses to await user input, similar to the Devin coding agent demo he references.

Before you watch

• Familiarity with basic RAG (retrieval-augmented generation: chunk, embed, retrieve, generate) is assumed for the second half of the talk.
• Watching the earlier lectures on LLM reasoning and ReAct-style agents helps, since both speakers reference reasoning and tool-use patterns introduced there without re-explaining them.

Check your understanding

1. In AutoGen, what are the two steps a developer follows to build a multi-agent application?
2. Why did splitting a task between a writer agent and a separate safeguard agent improve accuracy compared to a single agent doing both?
3. What are the four main limitations of a basic RAG pipeline that Jerry Liu identifies?
4. How does multimodal RAG index and retrieve information from a table or image that can't be directly embedded as text?
5. What is the trade-off between a constrained (router-based) agent flow and an unconstrained (ReAct-style) agent flow?