Lecture 5: Compound AI Systems and DSPy

What this lecture covers

This lecture asks why building an impressive language-model demo is easy but turning it into a reliable system is hard. It answers that question by introducing compound AI systems: programs that break a task into modular pieces, each handled by a language model or another component, rather than relying on one monolithic model. Examples include retrieval-augmented generation, multi-hop question answering, long-form report generation, and code-generation systems that search over many candidate solutions.

The lecture then argues that hand-written prompts make these systems fragile, because a single prompt string couples five different concerns at once. It presents DSPy as an alternative: a way to write compound AI systems as Python programs with declared natural-language modules, then let an optimizer generate and tune the actual prompts. After watching, you should understand what a compound AI system is, why prompt engineering doesn't scale, and how DSPy's signatures, modules, and optimizers address that.

Key ideas

• Compound AI system: a modular program where language models play specialized roles inside a larger architecture, instead of one model handling everything end to end.
• Coupling problem: a single hand-written prompt mixes five roles - specification, inference strategy, input formatting, objective, and model-specific coercion - which makes prompts break when you change the pipeline or the model.
• Signature: a declaration of a module's inputs and outputs in natural language, independent of how the prompt is phrased.
• Language model program: ordinary Python code (loops, conditionals) that calls fuzzy natural-language modules instead of hardcoded prompts.
• Adapter: the DSPy component that turns a signature into an initial, unoptimized prompt.
• Optimizer: an algorithm that treats prompts (and optionally weights) as tunable parameters and searches for settings that maximize a given metric.
• Bootstrapped demonstrations: examples generated by running a rough version of the program and keeping the input-output traces that scored well, used as few-shot examples.
• MIPRO: the optimizer discussed in depth, which bootstraps demonstrations, proposes instructions using a grounded description of the program, and uses a Bayesian surrogate model to search combinations of prompts across modules.

Walkthrough

Why compound AI systems matter (2:03)

The lecture opens by noting that language models are fluent enough to make errors hard to catch, and that a monolithic model is difficult to control, debug, or improve. Compound AI systems address this by decomposing a task into modular components. The lecture walks through several examples in increasing sophistication: retrieval-augmented generation for transparency, multi-hop retrieval for synthesizing information across pieces, the Storm system for generating long cited reports, and AlphaCodium for spending extra inference-time compute on code generation through reflection and ranking.

The coupling problem in hand-written prompts (9:07)

Each module in a compound system is typically implemented as a large hand-tuned prompt that bundles five things together: what the module should do (signature), how it should reason (inference strategy), how inputs are formatted and outputs parsed, what objective it should optimize for, and model-specific tricks needed to get a particular language model to comply. Because these are entangled, changing one module, swapping the language model, or adjusting the objective usually invalidates the whole prompt, which blocks portability and reuse.

DSPy: programs instead of prompts (13:10)

The lecture proposes writing compound systems as computer programs with embedded natural-language modules, drawing an analogy to compiling high-level code for different hardware. DSPy ("declaratively self-improving Python") lets you define a language model program as a Python class or function that takes natural-language inputs and calls modules declared only by their signature - what they take in and produce - without specifying how the model should do it. A worked multi-hop retrieval example shows a forward method looping to generate search queries and build context before producing an answer.

From signatures to prompts, and the role of optimizers (25:18)

An adapter turns a signature into a basic, unoptimized prompt. Optimizers then treat the whole program's prompts as parameters to tune against a metric, similar to choosing an optimizer like Adam in a neural network. The lecture shows results on a multi-hop QA task where adding retrieval, then optimization, substantially raises accuracy, and where a small optimized model can match or beat an unoptimized larger one. It also mentions that the same programs can be used to fine-tune small models via weight-based optimizers.

How DSPy optimizers work (30:20)

Most optimizers share a pattern: guess an initial prompt per module, use rejection sampling to collect input-output traces that score well according to a metric, and use those traces either as few-shot demonstrations, as material for a language model to induce better instructions, or as fine-tuning data. The lecture contrasts simple bootstrapped-demonstration search with prompt-optimization methods borrowed from single-prompt settings, such as OPRO, and explains why naively applying per-module coordinate ascent is expensive and doesn't co-optimize modules well.

MIPRO and grounded instruction proposal (48:31)

To generate better instructions, MIPRO grounds its proposals in context about the actual program: bootstrapped demonstration examples, a summary of the dataset, and a natural-language description of the whole pipeline built from inspecting the code. It then uses a Bayesian surrogate model, borrowed from hyperparameter optimization, to predict which combination of instructions and demonstrations across modules is likely to score well, evaluates candidates on a validation batch, and updates the surrogate model over repeated trials.

Benchmark results and real-world adoption (55:32)

Using the LangProBe benchmark, the lecture compares instruction-only optimization, demonstration-only optimization, and MIPRO's combination, finding that demonstrations tend to help more but instructions matter especially for tasks with conditional rules hard to convey through examples alone. It closes with case studies: a University of Toronto team using DSPy to win a medical QA competition by 20 points, a University of Maryland researcher outperforming 20 hours of manual prompt engineering in 10 minutes with DSPy, and production use at companies including JetBlue, Databricks, and Walmart.

Before you watch

• Familiarity with basic retrieval-augmented generation and prompting concepts (chain-of-thought, few-shot examples) is assumed.
• Basic Python and a general sense of how neural network training and optimizers work will help with the analogies used throughout.

Check your understanding

1. What are the five roles that a single hand-written prompt typically couples together, and why does that coupling limit portability?
2. How does a DSPy signature differ from a traditional prompt?
3. Describe the three general steps most DSPy optimizers follow when tuning a program.
4. Why does MIPRO ground its instruction proposals in a program description and dataset summary rather than just asking a model to generate instructions directly?
5. According to the benchmark results discussed, when does optimizing instructions tend to matter more than optimizing demonstrations?