Lecture 1: Introduction and Word Vectors

What this lecture covers

This opening lecture sets up the course, then asks a foundational question: how can a computer represent what a word means? It moves from the traditional dictionary-style view of meaning (a symbol paired with a thing) to a distributional view, where a word's meaning comes from the contexts it appears in. That shift motivates the rest of the course, which builds from word vectors up through neural networks, Transformers, and large language models.

By the end, you can explain why one-hot word representations fail to capture similarity, describe how word2vec turns a large corpus into dense word vectors by predicting nearby words, and follow the calculus behind the skip-gram objective function well enough to know what a training update to a word vector is actually doing.

Key ideas

• Word2vec: a 2013 algorithm (Mikolov et al.) that learns a vector for every word by using each word to predict the words around it in a text corpus.
• Distributional semantics: the idea, traced to linguist J.R. Firth, that a word's meaning is defined by the words that regularly occur near it, rather than by a fixed dictionary entry.
• One-hot vectors: a sparse representation where each word is an independent symbol; any two different words have a dot product of zero, so no similarity is encoded.
• Word embeddings: short, dense vectors (commonly 100 to 300 dimensions) where semantically related words end up close together and where directions in the space can carry consistent meaning.
• Center and outside words: word2vec's skip-gram setup predicts, for a chosen center word, the words in a fixed-size window around it.
• Softmax function: converts unbounded dot-product similarity scores into a probability distribution over possible context words.
• Gradient descent: word vectors start as random numbers and are repeatedly nudged in the direction that increases the model's predicted probability of the true context words.

Walkthrough

Course logistics and goals (2:07)

Professor Christopher Manning outlines the course: four assignments, a final project (default or custom), and a policy that assignments must be done individually, though AI tools may be used as coding aids rather than as answer generators. He states the three goals of the class: teaching deep learning methods for NLP from word vectors up to large language models, giving some grounding in the structure and difficulty of human language, and preparing students to actually build working NLP systems.

Human language, thought, and the rise of neural NLP (11:14)

Manning argues that language is what most separates humans from other intelligent species, both as a communication tool and as a scaffold for higher-level thought, with writing later extending that advantage across time and space. He then traces recent progress in NLP: neural machine translation becoming commercially viable around 2014 to 2016, search engines evolving from keyword matching to retrieval-and-synthesis "answer engines," GPT-2 in 2019 showing that models could generate fluent, coherent text, and finally chat-style large language models and multimodal "foundation models" that also generate images from text prompts.

From denotational to distributional meaning (28:28)

The lecture contrasts denotational semantics, where a word's meaning is the real-world thing it refers to (as encoded in resources like WordNet), with distributional semantics, where meaning comes from context. WordNet-style approaches are shown to miss nuance and slang and to be built by hand. One-hot vectors are introduced as the naive way to represent words numerically, and their key flaw is demonstrated: because every word occupies its own dimension, any two one-hot vectors are orthogonal, so there is no built-in notion that "hotel" and "motel" are related.

Word2vec: learning vectors by predicting context (46:50)

Word2vec is introduced as a simple, fast method for learning dense word vectors from a large corpus (a body of text). For every position in the text, the algorithm treats one word as the center word and looks at the words within a fixed window around it, trying to make the probability of the observed context words as high as possible. Sliding this process across the whole corpus and adjusting vectors accordingly is, in effect, the entire training procedure.

The skip-gram objective and the softmax (52:54)

The lecture formalizes this as an objective function: the average negative log likelihood of the observed context words, which the model minimizes via gradient descent. The probability of an outside word given a center word is defined using the dot product of their vectors, passed through the softmax function, which exponentiates the scores and normalizes them into a valid probability distribution. In this model, the only parameters are the word vectors themselves — one set for words acting as center words and one for words acting as outside words.

Computing the gradient by hand (1:05:09)

Manning works through the calculus of the skip-gram objective with respect to a center word vector, applying log rules and the chain rule to differentiate the softmax's numerator and denominator. The result takes an "observed minus expected" form: the actual outside word vector minus a probability-weighted average of all possible outside word vectors. This derivative is zero exactly when the model's predictions match what was actually observed in the text, which is the condition gradient descent is driving toward.

Before you watch

• No prior deep learning background is assumed, but basic linear algebra (vectors, dot products) makes the derivation easier to follow.
• Refreshing single-variable calculus, especially the chain rule and logarithm rules, helps with the gradient derivation section.

Check your understanding

1. Why do one-hot vectors fail to represent word similarity, and how does a dense word vector fix that?
2. In the skip-gram model, what exactly are the parameters being learned, and what role does the corpus play?
3. Why does word2vec use the softmax function instead of just using the raw dot product as a probability?
4. What does it mean for the gradient of the skip-gram objective to equal "observed minus expected," and why does that make sense for training?