Lecture 11: Benchmarking and Evaluation

What this lecture covers

The lecture asks a question self-learners rarely stop to consider: what does it actually mean to say a model is "good"? Yann Dubois walks through why evaluation looks different depending on whether you're training a model, tuning it, selecting between models, or deploying it in production, then works through the two broad families of NLP tasks: close-ended tasks with a small set of correct answers, and open-ended tasks like summarization, translation, and chat where many answers can be correct to different degrees.

It sits in the course after the modeling-heavy lectures on transformers and language models, and pairs naturally with later lectures on reasoning and agents, since those systems also need to be evaluated. After watching, you should be able to name the standard metrics for classification and generation tasks, explain why word-overlap metrics like BLEU and ROUGE are flawed, and understand why large language models are now commonly used as judges of other models' outputs.

Key ideas

• Evaluation changes by pipeline stage: metrics used during training must be fast, cheap, and differentiable; metrics used at deployment must be trustworthy, task-specific, and absolute rather than just comparative.
• Close-ended tasks: tasks like sentiment analysis, entailment, and question answering have a small number of correct answers and can be scored with standard machine learning metrics such as accuracy, precision, recall, and F1.
• Spurious correlations: models can score well on benchmarks like SNLI by exploiting artifacts in how humans wrote the data (for example, negation words correlating with "not entailed") rather than by understanding the task.
• Content-overlap metrics: BLEU and ROUGE compare generated text to a reference by counting shared n-grams; BLEU emphasizes precision, ROUGE emphasizes recall, and both miss semantic equivalence (yes versus heck yes).
• Model-based metrics: approaches like BERTScore and BLEURT compare learned representations instead of raw words, and BLEURT fine-tunes a pretrained model directly on human evaluation scores.
• Human evaluation is the gold standard but flawed: it is slow, expensive, suffers from inter- and intra-annotator disagreement, and crowd workers' incentives (getting paid quickly) don't always align with careful judgment.
• LLM-as-judge evaluation: asking a strong model like GPT-4 to compare two outputs, as in AlpacaEval and chatbot Arena, is far cheaper than human evaluation and correlates with human preferences about as well as humans agree with each other, though it inherits biases toward length and lists.
• Benchmark issues: multiple-choice benchmarks like MMLU can give different scores depending on prompt format and decoding method, and test-set contamination and overfitting make reported numbers hard to trust at face value.

Walkthrough

Why and when you measure performance (1:05)

The lecture opens with a mental model of the machine learning pipeline: training, hyperparameter tuning and development, model selection, and deployment. Each stage needs performance measurement, but with different requirements. Training needs a fast, differentiable loss; development still needs speed but should resist "shortcuts" that game the metric; deployment needs trustworthy, task-specific, absolute measures because there's no going back once a model is live. Academic publishing is treated as a separate case, where reproducibility and standardization across years of papers matter more than metric perfection, and where a benchmark must be hard enough to be interesting but not so hard that every model scores randomly.

Close-ended tasks and their standard benchmarks (8:09)

Close-ended tasks have a limited, enumerable set of correct answers, so they reduce to standard classification. The lecture surveys sentiment analysis (IMDB, SST), entailment (SNLI), part-of-speech tagging, coreference resolution, and question answering, then covers multitask suites like SuperGLUE that average scores across many tasks. It flags real pitfalls within this "simple" setting: choosing the wrong metric (accuracy is misleading on imbalanced data like spam detection), naively averaging metrics that measure different things, and models exploiting spurious correlations in how labels were collected rather than genuinely solving the task.

Open-ended tasks and content-overlap metrics (19:22)

Open-ended tasks such as summarization, translation, and instruction-following have many possible correct answers along a continuum of quality, so standard classification metrics don't apply. The lecture introduces content-overlap metrics: BLEU and ROUGE compare n-grams between a generated text and a human-written reference, with BLEU weighted toward precision (with a length penalty) and ROUGE toward recall. A worked example shows the weakness directly: a reference answer "heck yes" scores far higher against "yes" than against "yep," even though both convey the same meaning, and "heck no" scores deceptively high despite meaning the opposite.

Model-based metrics (25:25)

To move past literal word overlap, the lecture covers metrics built on learned representations: averaging word embeddings, then using contextual embeddings from BERT to compare generated and reference text (BERTScore), and finally BLEURT, which continues pretraining a BERT model to predict BLEU-like scores before fine-tuning it directly on human judgments. A separate finding complicates all reference-based metrics: reference summaries themselves (such as CNN/DailyMail's bullet points) are often low quality, and metrics like ROUGE-L correlate poorly with human judgment until the references are improved.

Human evaluation (33:30)

Human evaluation is presented as the gold standard for open-ended tasks and for validating automatic metrics, but the lecture spends most of this section on its practical problems: it is slow and expensive, suffers from low inter-annotator agreement (the AlpacaFarm team, despite writing detailed rubrics, only agreed 67% of the time), is not reproducible across papers (one study found only 5% of human evaluations were repeatable), and reflects annotator incentives such as speed over care, which can bias results toward longer or more detailed-looking answers.

Evaluating chatbots and current LLM benchmarks (43:39)

Because chatbots can be asked anything and produce arbitrarily long answers, they're typically evaluated by pitting two models against each other and asking which response is preferred, as chatbot Arena does with humans and Elo ratings. Using a strong model like GPT-4 as the judge, as in AlpacaEval, is roughly 100 times faster and cheaper than human evaluation and correlates as well with human preference as humans correlate with each other, though it requires correcting for biases toward longer, list-formatted, or self-generated answers. The lecture then surveys how current models are actually evaluated in practice: perplexity on held-out data (strongly correlated with downstream performance but not comparable across tokenizers or datasets), averaged scores across large benchmark suites like MMLU, HELM, and the Hugging Face Open LLM Leaderboard, and Arena-style pairwise comparison.

Challenges: consistency, contamination, and bias (1:03:57)

The closing section catalogs why current evaluation numbers deserve skepticism: multiple-choice results shift depending on answer-letter format and decoding method (three different implementations of MMLU gave meaningfully different scores for the same model), benchmarks can leak into pretraining data (contamination), and models can overfit to popular test sets over time. It also raises structural concerns: NLP benchmarking is dominated by English and by accuracy alone, aggregate metrics ignore compute cost and fairness, and using one LLM as a judge everywhere risks spreading that model's biases across the whole field. The lecture closes by urging researchers to read model outputs directly rather than trusting a single number.

Before you watch

• Be comfortable with standard classification metrics (accuracy, precision, recall, F1), since the lecture assumes this background for close-ended tasks.
• Some familiarity with what a language model's training objective (next-token prediction) looks like will help the perplexity discussion make sense.
• Knowing roughly what instruction-tuned chat models like ChatGPT are will help with the open-ended evaluation sections.

Check your understanding

1. Why does the lecture argue that a single "ideal" way of measuring performance doesn't exist across the machine learning pipeline?
2. What is the key difference between what BLEU and ROUGE emphasize, and why do both fail on the "heck yes" versus "yep" example?
3. Why did the AlpacaEval project need to reweight GPT-4's judgments, and what problem was it correcting for?
4. What does it mean that GPT-4's agreement with human preferences is higher than humans' agreement with each other, and why does the lecture treat this as a mixed result rather than pure good news?
5. What are two distinct ways a benchmark score can be inflated by "contamination" or overfitting, according to the lecture?