Lecture 7: Attention and Choosing a Final Project

What this lecture covers

This lecture answers two different questions. First, how do you evaluate a machine translation system automatically, and how does attention fix a fundamental weakness of the encoder-decoder RNN models from the previous lecture? Second, since final projects are due later in the course, how should you choose between a default project and a custom one, and what makes a strong project proposal?

The technical section covers the BLEU metric for scoring translations, then introduces attention: the idea that a decoder should look back at specific parts of the source sentence rather than relying on a single fixed-size hidden state. The second half is course guidance on selecting a topic, using compute resources, and writing a project proposal and paper. After watching, you should be able to explain why attention improves neural machine translation, walk through how attention scores and outputs are computed, and know what CS224N expects in a final project proposal.

Key ideas

• BLEU score: an automatic machine translation metric that scores overlap of 1- to 4-grams between a system's output and one or more human reference translations; it is imperfect because a sentence can be translated correctly in many different ways.
• Information bottleneck: in a plain encoder-decoder RNN, the entire meaning of the source sentence must be compressed into one fixed-size hidden state, which becomes a serious limitation for longer sentences.
• Attention: lets the decoder, at each generation step, compute a weighted combination of all the encoder's hidden states rather than relying only on the final one, so it can "look back" at relevant source words as needed.
• Attention score: a comparison, at each decoder step, between the decoder's current hidden state and every encoder hidden state; scores are passed through a softmax to get attention weights.
• Attention variants: dot-product attention is simplest; multiplicative (bilinear) attention inserts a learned matrix between the two vectors; additive attention uses a small feed-forward network; all three remain common ideas across later architectures.
• Attention as an interpretability tool: because attention weights show which source words the decoder focused on, they give a visual sense of what the model is aligning during translation.
• BERT default final project: CS224N's guided option, building and fine-tuning a minimal BERT implementation for sentiment analysis, with room to extend it.
• Custom final project: a self-directed research project that must substantively involve both human language and neural networks, requiring its own paper review, data, and evaluation plan.

Walkthrough

Evaluating machine translation with BLEU (2:05)

The lecture reviews how machine translation quality is measured. BLEU (bilingual evaluation understudy), proposed by IBM, compares a candidate translation against one or more human reference translations by counting overlapping 1- to 4-grams, with a penalty for translations that are too short. Because a sentence can be translated correctly in many different word orders and phrasings, BLEU is a crude proxy rather than a perfect measure, but it remains the most common automatic metric. Scores range from 0 to 100; getting into the 20s suggests the gist is understandable, while 30s and 40s indicate much better translations, and modern neural systems commonly reach the 50s and 60s, a sharp jump from the statistical translation systems that came before.

The bottleneck problem that motivates attention (12:18)

In the sequence-to-sequence model from the previous lecture, the encoder compresses an entire source sentence into a single final hidden state, which is the only information the decoder receives. The lecture argues this is implausible for longer sentences and unlike how a human translator works, since a person re-reads earlier parts of a sentence while translating rather than relying purely on memory. This observation, from the original 2014-2015 neural machine translation work, motivated giving the decoder direct access to every encoder hidden state instead of just the last one.

How attention works (15:25)

At each decoder step, the current decoder hidden state is compared against every encoder hidden state to produce a set of attention scores, most simply by taking a dot product between each pair of vectors. These scores are passed through a softmax to produce attention weights, and the encoder hidden states are combined into a weighted average using those weights, called the attention output. This output is concatenated with the decoder's hidden state and used to predict the next output word. The process repeats at every decoding step, so the model can attend to different source words as it generates each word of the translation, for example focusing on the matching source word when translating "he," "hit," and "me" in sequence.

Attention's impact and variants (21:30)

Attention proved transformative: a 2014 Google system used a very large, deep pure LSTM without attention, while a nearly contemporaneous University of Montreal system using attention got better results with far less compute. Attention solves the bottleneck problem, provides shortcut connections that help with vanishing gradients similar to residual connections, and gives a degree of interpretability by showing which source words the model attended to at each step. The lecture covers three ways to compute attention scores: simple dot-product attention; multiplicative (bilinear) attention, which inserts a learned matrix between the two hidden-state vectors so their dimensions do not need to align directly, and can be made more efficient using a low-rank factorization; and additive attention, which runs the two vectors through a small feed-forward network. All three ideas recur in later architectures, including the dot-product-style attention used in Transformers.

Choosing a final project (37:46)

The course allows teams of one to three, with bigger teams expected to do proportionately more work; grading accounts for team size in judging whether the project's scope was appropriate. The default final project is a guided BERT implementation, where students complete and fine-tune a minimal BERT for sentiment analysis and then extend it with an idea such as paraphrasing, contrastive learning, or low-rank adaptation, with a leaderboard for feedback. A custom final project is self-directed and must substantively involve both human language and neural networks, though it can combine other modalities like vision or audio. The lecture also covers practical compute resources: limited free credits on cloud providers, notebook services like Google Colab and Kaggle, and API credits for working with large language models rather than training models from scratch.

Writing the proposal and finding a topic (53:03)

A project proposal requires a two-page critical review of a key research paper, a description of the planned approach, data, and evaluation method, and a new requirement to discuss potential ethical considerations. The lecture stresses having an appropriate baseline to compare against, and lists several project types: applying NLP to solve a task, testing a new neural network idea, using large language models via in-context learning or fine-tuning, and less common options like interpretability or theoretical analysis projects. It recommends looking at past CS224N projects, the ACL Anthology, and conference proceedings for ideas, while noting that in the current era, most projects realistically build on existing pre-trained models rather than training new architectures from scratch, since the compute required for state-of-the-art results is usually out of reach for a class project.

Before you watch

• Watch the prior lecture on RNN language models, LSTMs, and encoder-decoder machine translation, since attention is introduced as a direct fix to that architecture's limitations.
• Be comfortable with softmax and weighted averages, since attention scores and outputs are built from them.
• If you are enrolled in the course, review the syllabus sections on assignments and grading before this lecture's project-planning content, since it assumes you already know the basic project timeline.

Check your understanding

1. Why is a single reference translation less reliable for BLEU scoring than multiple references, and what does BLEU actually measure?
2. What specific limitation of the encoder-decoder RNN from the previous lecture does attention address, and how?
3. Walk through the steps used to compute an attention output at one decoder time step, from hidden states to the final weighted average.
4. What is the difference between dot-product, multiplicative, and additive attention, and what problem does the low-rank factorization of multiplicative attention solve?
5. What must a CS224N custom final project substantively involve, and what are the two required parts of a final project proposal?