Lecture 11: Natural Language Generation

What this lecture covers

This lecture asks how systems that produce fluent text, from machine translation to open-ended story generation, actually decide what to write and how to check whether they write it well. It follows CS224N's earlier lectures on language modeling and sequence-to-sequence architectures, and extends them into three practical concerns: how to pick tokens from a model's output distribution (decoding), how training choices like teacher forcing create problems at generation time (exposure bias), and how to measure whether generated text is actually good.

By the end, you should be able to distinguish open-ended from non-open-ended generation tasks, explain why greedy or beam search decoding causes repetition and how sampling methods like top-k, top-p, and temperature address it, describe exposure bias and the main fixes for it, and compare content-overlap, model-based, and human evaluation methods. The lecture closes with the ethical risks of deploying generation models, including toxicity and factual unreliability.

Key ideas

• Open-endedness spectrum: NLG tasks range from non-open-ended (machine translation, summarization), where the input nearly determines the output, to open-ended (story generation, chit-chat), where many valid outputs exist.
• Autoregressive generation: a language model predicts one token at a time from a score distribution, feeding each predicted token back in to predict the next.
• Teacher forcing vs. exposure bias: models are trained on gold previous tokens but must generate from their own previous predictions at test time, which creates a training-test mismatch.
• Maximum-probability decoding fails for open-ended text: greedy decoding and beam search tend to produce repetitive, self-amplifying text even in very large models.
• Truncated sampling: top-k sampling restricts choices to the k highest-probability tokens, and top-p (nucleus) sampling restricts choices to a cumulative probability mass, adapting better to how flat or peaked the distribution is.
• Temperature: rescaling scores before softmax makes the output distribution flatter (more diverse) or spikier (closer to greedy) without changing token rankings.
• Reward-based training: reinforcement learning, including learning from human preference data, can align generation with objectives beyond maximum likelihood, and underlies systems like ChatGPT.
• Evaluation hierarchy: content-overlap metrics like BLEU and ROUGE are cheap but miss semantic meaning, model-based metrics like BERTScore and MAUVE capture more semantics, and human judgment remains the gold standard despite being slow and inconsistent.

Walkthrough

What is NLG and the open-endedness spectrum (2:07)

The lecture defines natural language generation as the branch of NLP where the task output is natural language, covering machine translation, dialogue systems, summarization, creative writing, data-to-text, and image captioning. It introduces a spectrum of open-endedness: machine translation and summarization sit near the non-open-ended end because the input strongly constrains valid outputs, while story generation and chit-chat dialogue sit near the open-ended end because many different continuations are valid. This distinction, which can be formalized via output entropy, determines which architectures and decoding methods work well later in the lecture.

Reviewing autoregressive generation and architecture choices (8:12)

The lecture reviews how an autoregressive model computes a score for each vocabulary token at every step, converts it to a probability distribution with softmax, and predicts tokens left to right, feeding each prediction back in. It connects this to architecture choice: encoder-decoder models suit less open-ended tasks like machine translation, while decoder-only models are more common for open-ended generation, though the choice is more about compute allocation than a strict requirement. Training via maximum likelihood is reintroduced as a per-step classification problem, using teacher forcing during training.

Decoding algorithms and the repetition problem (15:25)

Greedy decoding and beam search both search for high-probability strings, which works reasonably for non-open-ended tasks but produces highly repetitive text for open-ended generation. The lecture shows that repeated phrases become progressively more probable under the model, a self-amplification effect that persists across architectures and even at very large model scale. Fixes discussed include n-gram blocking, which forbids repeating a previously seen n-gram, and alternative training objectives such as unlikelihood training and coverage loss that discourage repetition directly.

Sampling-based decoding: top-k, top-p, and temperature (22:28)

Since maximum-probability decoding does not match how humans write, the lecture introduces sampling from the model's distribution instead. Plain sampling risks selecting from a long tail of low-probability but numerous bad tokens, so top-k sampling restricts sampling to the k highest-probability tokens, and top-p (nucleus) sampling instead samples from the smallest set of tokens whose cumulative probability exceeds p, which adapts automatically to how flat or peaked the distribution is. The temperature parameter is introduced as a way to rescale scores before softmax, flattening the distribution for more diversity or sharpening it toward greedy decoding. The section ends with re-ranking: generating several candidate sequences and scoring them, for example with perplexity or task-specific scoring functions, to select the best one.

Training language models: exposure bias and its fixes (40:42)

The lecture connects the repetition problem back to training: models trained with teacher forcing see only gold previous tokens during training but must condition on their own, possibly flawed, previous predictions at test time, a mismatch called exposure bias. Proposed remedies include scheduled sampling, which gradually mixes in model-generated tokens during training, dataset aggregation (DAgger), which periodically adds model-generated sequences to the training data, and retrieval-augmented generation, which edits a retrieved high-quality prototype instead of generating purely left to right. The lecture also introduces reinforcement learning framings of generation, where the reward can come from evaluation metrics or from a model trained on human preference data, the technique underlying ChatGPT's pipeline of pretraining, instruction tuning, and RLHF.

Evaluating NLG: content overlap, model-based, and human metrics (54:53)

Three families of evaluation are compared. Content-overlap metrics like BLEU and ROUGE are fast but rely on lexical overlap, so they can penalize correct paraphrases and reward wrong answers that happen to share words; they work reasonably for non-open-ended tasks like machine translation but get progressively worse for more open-ended tasks. Model-based metrics such as BERTScore, word mover's distance, and MAUVE use embeddings to capture semantic similarity, and MAUVE specifically compares the distribution of generated text to human text in a discretized embedding space, making it usable for open-ended generation. Human evaluation remains the gold standard for correlating with human judgment, but it is slow, expensive, inconsistent between annotators, and measures precision rather than recall over the space of good outputs.

Ethical considerations of deployed generation models (1:12:03)

The lecture closes on risks from deploying NLG systems: generating harmful or toxic content, being manipulated through adversarial prompts, and stating factual errors fluently and convincingly. Because generation models are built on language models pretrained on internet data, they can reproduce biases and stereotypes present in that data, and full data cleaning is impractical at scale. The lecture stresses that models should not be deployed without safeguards and testing against adversarial triggers, and that the field still needs better automatic evaluation and more careful consideration of how users interact with these systems.

Before you watch

• Review the CS224N lectures on language modeling and machine translation, since this lecture assumes familiarity with autoregressive decoding, encoder-decoder architectures, and teacher forcing.
• Recall how softmax converts scores into a probability distribution over the vocabulary, since decoding and temperature both operate on this distribution.
• Some familiarity with reinforcement learning basics (states, actions, policies, rewards) helps with the training section, though the lecture only summarizes it.

Check your understanding

1. Why does maximum-probability decoding (greedy or beam search) tend to produce repetitive text in open-ended generation, and why does increasing model scale not fix this?
2. How does top-p (nucleus) sampling adapt differently than top-k sampling when the model's output distribution is flat versus peaked?
3. What is exposure bias, and how do scheduled sampling and dataset aggregation each try to reduce it?
4. Why do content-overlap metrics like BLEU perform reasonably for machine translation but poorly for open-ended tasks like story generation?
5. What are the main drawbacks of human evaluation as the gold standard for judging generated text?