Lecture 2: Word Vectors and Language Models

What this lecture covers

This lecture completes the picture of word vectors that Lecture 1 started, then begins the transition toward neural networks proper. It closes the loop on how word2vec is actually trained with stochastic gradient descent, demonstrates the semantic properties of trained vectors (including the famous king − man + woman = queen analogy), and introduces a faster training trick called negative sampling.

It then pivots to an alternative way of building word vectors from raw co-occurrence counts, which leads into the GloVe algorithm developed at Stanford, and discusses how word vectors are evaluated. The lecture ends by tackling the problem of words with multiple meanings and taking the first steps toward neural classifiers, setting up the neural network material that continues in Lecture 3.

Key ideas

• Stochastic gradient descent (SGD): instead of computing the exact gradient over an entire dataset, SGD estimates it from a small random batch of examples, which is far faster and, in neural networks, often works better because of the added noise.
• Word vector analogies: arithmetic on trained word vectors (such as king − man + woman) lands near semantically related words (queen), showing that word2vec captures linear meaning components, not just similarity.
• Negative sampling: a faster alternative to the full softmax that trains a word vector by contrasting the true context word against a handful of randomly sampled "negative" words, using the logistic (sigmoid) function instead of softmax.
• Co-occurrence matrix: a matrix of how often each word appears near every other word in a corpus; on its own it is far higher-dimensional than word vectors, so it needs dimensionality reduction.
• GloVe: a Stanford model that builds word vectors so that the log ratio of co-occurrence probabilities between words corresponds to a linear direction in vector space.
• Intrinsic vs. extrinsic evaluation: intrinsic evaluation scores a component directly (such as word similarity or analogy accuracy), while extrinsic evaluation measures whether it improves a downstream task, such as named entity recognition.
• Word senses: most words (like "pike" or "bank") have several meanings; a single word vector ends up as a frequency-weighted average of those senses, sometimes described as a superposition.

Walkthrough

Finishing gradient descent and introducing SGD (4:08)

Manning completes the loop on basic gradient descent: compute the gradient of the loss function, take a small step in the downhill direction scaled by a learning rate, and repeat. He explains that computing the exact gradient over a full dataset is too slow in practice, so neural networks instead use stochastic gradient descent, estimating the gradient from a small mini-batch (for example 16 or 32 examples) at each step. He also notes that word vectors must be initialized with small random numbers, not zeros, or the model gets stuck due to symmetry.

Exploring trained word vectors and analogies (11:17)

Using a Jupyter notebook with pretrained GloVe vectors loaded through the gensim package, Manning shows that words like "bread" and "croissant" have visibly similar vector components, and that querying for words most similar to "USA" or "banana" returns sensible neighbors. He demonstrates the vector-arithmetic analogy trick — man is to king as woman is to (queen) — and runs several playful examples (Australia is to beer as Russia is to vodka), illustrating that the model captures cultural associations well beyond narrow dictionary meaning.

Skip-gram with negative sampling (27:31)

Because the full softmax requires summing over the entire vocabulary (hundreds of thousands of words) for every training step, the original word2vec paper proposes negative sampling as a cheaper alternative. Instead of scoring every possible context word, the model trains simple logistic classifiers: it pushes up the probability of the true context word and pushes down the probability of a few randomly sampled negative words, using the sigmoid function. Negative words are sampled from a unigram distribution raised to the power of 3/4, which slightly boosts the chances of rarer words being sampled compared to sampling by raw frequency.

Co-occurrence counts and the GloVe algorithm (34:44)

As an alternative to word2vec's prediction-based approach, the lecture builds a small co-occurrence count matrix from a toy corpus, where each cell counts how often two words appear within a context window of each other. This matrix is far too large and sparse to use directly, so dimensionality reduction (related to singular value decomposition) is needed. Manning connects this to earlier work by Doug Rohde and then explains the key idea behind GloVe, developed with Jeffrey Pennington: the ratio of co-occurrence probabilities between words carries meaning, and taking a log turns that ratio into something linear, so GloVe fits a model where the dot product of two word vectors approximates the log of their co-occurrence probability.

Evaluating word vectors (45:57)

The lecture distinguishes intrinsic evaluation, such as scoring analogy accuracy or comparing model similarity scores to human similarity judgments, from extrinsic evaluation, such as measuring whether adding word vectors improves accuracy on a downstream task like named entity recognition. Both evaluation types are shown to favor GloVe and word2vec over older approaches like plain SVD on raw counts.

Word senses and the first neural classifier (52:01)

Manning discusses how words like "pike" or "bank" carry multiple senses, and how a single trained word vector ends up as a weighted average across those senses rather than a separate vector per sense, since senses are often not cleanly separable. He connects this back to a sparse-coding technique that can approximately recover individual sense vectors from the combined one. The lecture then introduces a simple neural classifier for named entity recognition (deciding whether a word like "Paris" refers to a location), building a small window-based network with a hidden layer and a logistic output, foreshadowing the fuller treatment of neural networks in the next lecture.

Before you watch

• Watch Lecture 1 first: this lecture assumes you already know how word2vec's skip-gram objective and softmax work.
• A basic grasp of probability distributions and logarithms helps with the negative sampling and GloVe sections.

Check your understanding

1. Why is stochastic gradient descent preferred over full-batch gradient descent for training neural networks?
2. What problem does negative sampling solve, and how does it avoid computing a full softmax?
3. What is the intuition behind using the log ratio of co-occurrence probabilities in the GloVe model?
4. Why can't a single word vector cleanly separate a word's different senses, and what does the vector represent instead?
5. What is the difference between an intrinsic and an extrinsic evaluation of word vectors, and give an example of each.