Lecture 5: Language Models and Recurrent Neural Networks

What this lecture covers

This lecture asks how a system can predict which word comes next in a piece of text, and how that simple task became the foundation of modern NLP. It opens with a short set of practical neural-network training techniques that assignment 2 relies on, then defines what a language model is and traces its history from classical n-gram counting to neural approaches.

The core of the lecture introduces recurrent neural networks (RNNs) as a language model that processes text of any length with a fixed set of weights, and walks through how such a model is trained and used to generate new text. After this lecture you should be able to explain what a language model estimates, build a simple n-gram model by counting, and describe how an RNN updates a hidden state word by word to produce a probability distribution over the next word.

Key ideas

• Language model: a system that assigns a probability distribution over the next word given preceding context, or equivalently assigns a probability to any piece of text via the chain rule.
• N-gram model: predicts the next word using counts of the preceding n-1 words, making a Markov assumption that only recent context matters.
• Sparsity and back-off: n-gram counts hit zero for unseen sequences, so practitioners smoothed counts and backed off to shorter contexts when a longer one had no data.
• Fixed-window neural language model: concatenates word embeddings for a fixed number of prior words and feeds them through a hidden layer to predict the next word, avoiding the storage blow-up of n-grams but still limited by a fixed window.
• Recurrent neural network: reuses the same weight matrices at every time step, updating a hidden state that summarizes everything read so far, so it can use context of any length with a constant-size model.
• Teacher forcing: during training, the model is always fed the actual next word from the text rather than its own generated guess, which keeps training well-defined.
• Dropout: randomly zeroes some activations during training to prevent features from relying too heavily on each other, acting like training an ensemble of sub-networks.
• Modern overfitting view: large neural networks can fit training data almost perfectly and still generalize well, unlike the classical picture where training loss and test loss diverge.

Walkthrough

Neural network training tricks (7:16)

The lecture opens with several practical techniques used in assignment 2. It reframes regularization: the classical view says small weights prevent overfitting, but modern large networks are trained to near-zero training loss and still generalize well, so regularization instead helps huge models find averaging solutions that work on new data. Dropout is presented as the main regularization technique used in deep learning: at training time a random mask zeroes out some activations in each pass, forcing the network to not over-rely on any single feature; at test time nothing is dropped but activations are rescaled. The lecture also covers vectorization (avoiding for-loops in favor of matrix and tensor operations for speed), the need for small random parameter initialization to break symmetry, and optimizers, noting that Adam is a safe default that adapts step sizes per parameter based on past gradients.

What a language model is (24:36)

A language model is defined as a system that predicts a probability distribution over the next word given a context, or equivalently, via the chain rule, assigns a probability to any full piece of text. The lecture notes that language models have been central to NLP since at least the 1980s, powering everything from phone keyboard suggestions to search-query completion, long before large chatbot systems existed.

N-gram language models (28:42)

From 1975 to around 2012, language models were mostly n-gram models: they count how often short word sequences occur in a corpus and use a Markov assumption to predict the next word from only the preceding n-1 words. The lecture works through an example using 4-grams, shows how counts convert directly into probability estimates, and discusses the two core problems: sparsity (many valid continuations never appear in training data, giving zero probabilities that were fixed by smoothing counts and backing off to shorter contexts) and storage cost (the number of distinct n-grams grows exponentially with context size, which is why 5-grams were about the practical limit). Despite their simplicity, n-gram models can generate surprisingly grammatical, if incoherent, text, and the lecture draws a parallel between the old "just collect more text" scaling strategy and today's emphasis on scaling neural models.

From n-grams to neural language models (43:59)

A fixed-window neural language model replaces n-gram counting with a neural network: word embeddings for a fixed number of preceding words are concatenated and passed through a hidden layer and a softmax to predict the next word. This removes the sparsity and storage problems of n-grams, but it still uses only a fixed window of context, and the same word is processed by different parameters depending on its position in that window, which is an inefficient way to represent the same information.

Recurrent neural networks (50:06)

RNNs solve the fixed-window and positional-parameter problems. A single weight matrix is applied at every time step: at each step, the previous hidden state and the current word embedding are each multiplied by their own weight matrix, summed, passed through a bias and a nonlinearity (commonly tanh), and produce a new hidden state. Because the same weights are reused everywhere, the model size stays constant regardless of context length, and information from any earlier word can in principle influence the current prediction. The main practical drawback is that this sequential computation is slow, since each hidden state depends on the one before it, and in practice older information tends to fade from the hidden state faster than desired.

Training and generating text with RNNs (58:13)

RNN language models are trained with cross-entropy loss: at each position, the model predicts a distribution over the next word and is penalized by the negative log probability it assigned to the actual next word, using teacher forcing so training always continues from the true text rather than the model's own guesses. Because full corpora are far too long to backpropagate over directly, training data is cut into fixed-length segments (often simply 100 words), and gradients for the repeated weight matrix are computed by summing contributions from every time step, sometimes truncated to a shorter window for speed. To generate text, the model starts from a start-of-sequence symbol, samples a word from the predicted distribution, feeds that word back in as the next input, and repeats until an end-of-sequence symbol appears. The lecture closes with playful examples: RNNs trained on Obama speeches, Harry Potter text, recipes, and paint-color names, showing both the fluency and the limits of this simple architecture.

Before you watch

• Be comfortable with how a basic feed-forward neural network computes a prediction from concatenated word embeddings, covered in earlier lectures on word vectors and classifiers.
• Know what cross-entropy loss and softmax do, since the language modeling objective reuses them directly.
• Review backpropagation and the chain rule, since the lecture builds on how gradients flow through repeated computations.

Check your understanding

1. Why does a classical n-gram language model run into a sparsity problem, and what are two ways the lecture describes to work around it?
2. What specific limitation of fixed-window neural language models does a recurrent architecture remove, and how does it do so?
3. What is teacher forcing, and why is it used when training an RNN language model instead of letting the model generate freely?
4. Walk through the steps an RNN takes to generate a new sentence once it is trained, from the start symbol to the end symbol.
5. Why are large modern neural networks trained to near-zero training loss without this being treated as overfitting?