Lecture 12: Evaluation

What this lecture covers

Evaluation looks mechanically simple: run a model, get scores, average them. This lecture argues it is anything but simple, because there is no single correct evaluation, only ones suited to a specific question, whether that question comes from a user choosing a product, a researcher tracking scientific progress, or a policymaker weighing benefits and harms. The lecture works through a framework for thinking about evaluation design (inputs, how the model is called, how outputs are assessed, how results are interpreted), then surveys perplexity, knowledge and reasoning benchmarks, instruction-following evaluations, agent benchmarks, and safety benchmarks, pointing out the gaming, contamination, and validity problems that affect each.

This is lecture 12 of Stanford's CS336. After watching, you should be able to explain what perplexity measures and why it is still useful despite the shift to task-based evaluation, describe how benchmarks like MMLU, GPQA, and Humanity's Last Exam differ in difficulty and construction, and identify common failure modes in evaluation: train-test contamination, benchmark gaming, and mismatches between what a benchmark measures and what people actually care about.

Key ideas

• No single true evaluation: the right evaluation depends on the question being asked (purchase decision, scientific progress, policy, model development), so a good score on one benchmark does not automatically answer a different question.
• Perplexity: measures how well a model's probability distribution matches a reference text distribution; it is smoother and less gameable than task accuracy, and still central to scaling law fits.
• Train-test contamination: because models train on huge scraped web corpora, verifying that a test set was never seen during training is difficult; deduplication by n-gram overlap catches only some cases.
• Benchmark saturation and succession: benchmarks like MMLU get replaced by harder versions (MMLU-Pro, GPQA, Humanity's Last Exam) once frontier models approach ceiling performance, but each new benchmark introduces its own construction biases.
• LLM-as-judge evaluation: benchmarks like AlpacaEval and WildBench use a language model to score open-ended responses, which is efficient and automatic but biased and gameable (for example, favoring longer responses).
• Agent benchmarks: tasks like SWE-bench, Cybench, and MLE-bench evaluate a model plus scaffolding acting over multiple steps with tools, and accuracies on these remain low, showing a real capability gap versus single-turn benchmarks.
• Capability versus propensity: capability is whether a model can do something at all; propensity is whether it is willing to. Safety evaluation and jailbreaking research target propensity, while open-weight models raise capability concerns because fine-tuning can remove safety training.

Walkthrough

The evaluation crisis (0:05)

The lecture opens with a barrage of competing signals used to judge models: benchmark scores like MMLU and GPQA, cost-adjusted leaderboards like Artificial Analysis, usage-based rankings like OpenRouter, pairwise preference rankings like Chatbot Arena, and informal "vibes" from social media examples. It frames this as an "evaluation crisis": popular benchmarks like MMLU may be saturated or gamed, and there is no consensus on which signals to trust, setting up the rest of the lecture's more careful treatment.

A framework for evaluation design (5:11)

Evaluation is broken into questions about inputs (what prompts to use, whether they cover rare or difficult cases, whether they should be adapted to the specific model being tested), how the model is called (zero-shot, few-shot, chain-of-thought, tool use, and whether the object of evaluation is the model or the full system including scaffolding), how outputs are assessed (exact match versus other metrics, whether cost is factored in, how to score open-ended generation), and how results are interpreted (what a given score actually implies, and whether train-test overlap undermines it). The lecture stresses that each choice depends on the underlying goal of the evaluation.

Perplexity: still useful despite the shift to tasks (16:22)

Perplexity measures how well a model assigns probability to a held-out dataset, and was the dominant evaluation metric in 2010s language modeling research using datasets like Penn Treebank and WikiText. GPT-2 and GPT-3 shifted the field toward zero-shot or few-shot evaluation on downstream tasks, but the lecture argues perplexity remains valuable: it is smoother than accuracy, less gameable when train and test sets are properly separated, and central to fitting scaling laws. The lecture also flags a practical risk: because computing perplexity requires trusting a provider's reported probabilities, it is easier to get wrong or manipulate than a straightforward accuracy check on generated text.

MMLU and its harder successors (32:40)

MMLU, introduced in 2020, tests knowledge across 57 subjects with multiple-choice questions scraped from the web, originally evaluated with few-shot prompting on base models before instruction tuning existed. The lecture notes MMLU tests knowledge more than "language understanding" despite its name, and that its ceiling has been reached by frontier models, prompting successors: MMLU-Pro (ten answer choices instead of four, noisy questions removed), GPQA (PhD-level, "Google-proof" questions validated by experts), and Humanity's Last Exam (crowdsourced hard multimodal questions with a prize pool for creators). Each newer benchmark shows lower scores initially but gets progressively "solved" as models improve, and each has construction biases worth being skeptical of.

Instruction following and LLM-as-judge evaluation (52:00)

Since instruction-following models can be asked to do open-ended, one-off tasks rather than fixed benchmark formats, evaluating them is harder. Chatbot Arena collects live pairwise human preferences and computes Elo-style rankings, but has faced criticism (the "leaderboard illusion" paper) over unequal access for some providers. IFEval checks only whether synthetic formatting constraints (word counts, required words, no commas) are satisfied, not response quality. AlpacaEval and WildBench use a language model as judge to score responses, which is fast and automatic but was shown to favor longer responses until a length-corrected variant was introduced.

Agent benchmarks reveal a real capability gap (1:00:07)

Agent benchmarks test a model plus scaffolding acting iteratively with tools over time: SWE-bench asks an agent to fix a GitHub issue so unit tests pass, Cybench asks an agent to solve capture-the-flag cybersecurity challenges (with the human "first solve time" recorded as a difficulty signal), and MLE-bench asks an agent to complete Kaggle-style machine learning competitions end to end. Accuracies on these tasks remain well below 20-25% for most models, showing that current models handle isolated question-answering far better than sustained, tool-using work. The lecture also briefly covers ARC-AGI, which strips away world knowledge and language to test pure pattern-based reasoning, where models historically scored near zero until recent reasoning models made meaningful progress at high compute cost.

Safety benchmarks and evaluation validity (1:06:15)

Safety benchmarks like HarmBench and AIR-Bench test whether models refuse harmful requests, but the lecture cautions that a high refusal rate alone is meaningless without a paired capability evaluation, since a model that refuses everything looks "safe" but is useless. Jailbreaking research shows that optimized prompts can bypass safety training even in models with strong refusal behavior. The lecture distinguishes capability (can the model do something) from propensity (will it), noting open-weight models pose different risks since fine-tuning can strip away safety training. It closes on broader validity concerns: standardized benchmarks are often unrealistic compared to real usage, train-test contamination is essentially unverifiable at web scale, and many popular benchmarks (including MATH and GSM8K) contain nontrivial label noise that inflates or deflates reported scores.

Before you watch

• No specific prior lecture is required, but familiarity with basic language model training (from earlier lectures) helps when the discussion turns to perplexity and train-test overlap.
• It helps to have seen at least one benchmark leaderboard (such as MMLU or Chatbot Arena) beforehand, since the lecture assumes some familiarity with what these numbers typically look like.

Check your understanding

1. Why does the lecture argue there is no single "true" evaluation of a language model?
2. What does perplexity measure, and why is it harder to trust than a straightforward accuracy score?
3. How do MMLU-Pro, GPQA, and Humanity's Last Exam each try to address the saturation of MMLU, and what limitation does each introduce?
4. What is the difference between capability and propensity in the context of AI safety, and why does it matter differently for API-only versus open-weight models?
5. Why do agent benchmarks like SWE-bench and MLE-bench show much lower accuracy than single-turn question-answering benchmarks?