Lecture 4: Dependency Parsing

What this lecture covers

Switching from math to linguistics, this lecture asks how a sentence's structure — which words modify which — can be represented and predicted automatically. It contrasts constituency (phrase structure) grammar with dependency grammar, then spends significant time showing, through real ambiguous newspaper headlines, why human language structure cannot be resolved by rules alone the way programming language syntax can.

The second half builds toward the actual parser assignment: it covers how annotated treebanks replaced hand-written grammar rules, how transition-based parsing turns parsing into a sequence of shift/reduce decisions, and how a neural network can replace older symbolic classifiers to make those decisions faster and more accurately. After this lecture you can describe a sentence's dependency structure, walk through the shift-reduce parsing algorithm by hand, and explain what a neural transition-based parser predicts at each step.

Key ideas

• Constituency (phrase structure) grammar: builds sentence structure by grouping words into nested phrases (noun phrase, prepositional phrase, verb phrase) using rules similar to a context-free grammar.
• Dependency grammar: represents structure as directed arcs from head words to the words that modify or depend on them, rather than nested phrases.
• Global syntactic ambiguity: unlike programming languages, human language sentences can have several fully valid readings (for example, a series of prepositional phrases can attach in many different ways) that only real-world context resolves.
• Treebanks: large collections of hand-parsed sentences (such as Universal Dependencies) that replaced hand-written grammar rules starting in the 1990s, providing both training data and a way to evaluate parsers.
• Transition-based parsing: builds a dependency parse by moving words from a buffer onto a stack and repeatedly choosing one of three actions — shift, left-arc, or right-arc — until the sentence is fully parsed.
• UAS and LAS: unlabeled attachment score and labeled attachment score, the standard ways to measure what fraction of a parser's predicted dependency arcs (and their labels) match a gold-standard parse.
• Neural transition-based parser: replaces sparse, hand-crafted indicator features with dense word, part-of-speech, and label embeddings fed through a small neural network, giving both higher accuracy and much faster parsing.

Walkthrough

Constituency grammar: phrases built from parts of speech (2:06)

Manning introduces parts of speech (nouns, adjectives, determiners, prepositions) and shows how they combine into larger units like noun phrases and prepositional phrases, which can themselves combine into bigger phrases. This structure can be captured with a context-free grammar, giving a first way to formally describe the structure of English sentences.

Dependency grammar: heads and their dependents (9:14)

As an alternative representation, dependency grammar identifies a head word (such as the verb "look" in "look in the large crate in the kitchen by the door") and draws arrows to the words that modify it, directly or indirectly. Manning argues that since spoken and written language arrive as a flat linear stream with no inherent structure markers, listeners and models alike must actively work out what modifies what to recover meaning.

Why human language is globally ambiguous (15:19)

Using real newspaper headlines ("Scientists count whales from space," a Wall Street Journal sentence with four stacked prepositional phrases, and several other examples), the lecture shows that sentences often have multiple fully grammatical readings, with the number of possible attachments growing according to the Catalan number series as more prepositional phrases are added. Unlike programming languages, where ambiguity is always resolved by fixed rules, human language relies on listeners using context and world knowledge to pick the intended reading.

The history and vocabulary of dependency grammar (29:31)

The lecture formalizes dependency arcs, their labels (such as subject or prepositional modifier), and the convention of adding a fake root node at the top of the tree. It traces dependency grammar back to Panini's grammar of Sanskrit, composed and transmitted orally roughly two and a half millennia ago, contrasting this long history with the comparatively recent (1940s-50s) development of phrase-structure and Chomsky hierarchy formalisms.

From hand-written rules to annotated treebanks (38:43)

Early NLP tried to parse language using hand-written grammar rules and dictionaries, but this approach struggled with the sheer creativity and messiness of real language use and offered no principled way to rank competing parses. Starting in the late 1980s and through the 1990s, the field shifted toward building large annotated treebanks, such as the multilingual Universal Dependencies project, which provide statistics for making parsing decisions and, crucially, a standard way to evaluate how accurate a parser is.

Transition-based parsing: the shift-reduce algorithm (54:56)

Manning walks through a concrete worked example, parsing "I ate fish," using a stack (initialized with a root symbol) and a buffer holding the sentence's words. At each step the parser chooses one of three actions: shift (move the next buffer word onto the stack), left-arc, or right-arc (pop the top two stack items and record one as a dependent of the other). This produces a full dependency tree in linear time, in contrast to the more expensive dynamic-programming methods used for context-free grammars.

Neural transition-based parsing and graph-based alternatives (1:07:14)

The lecture contrasts older symbolic parsers, which relied on huge numbers of sparse, hand-crafted indicator features and were evaluated with UAS and LAS, with a neural transition-based parser that instead concatenates dense embeddings for words, parts of speech, and dependency labels, feeding them through a small neural network to predict shift, left-arc, or right-arc. This approach, developed at Stanford and later scaled up by Google as "Parsey McParseface," matched the accuracy of slower graph-based parsers while remaining fast. The lecture closes by briefly describing graph-based dependency parsing, which scores every possible head for every word and finds a minimum spanning tree, noting that a neural graph-based parser is used in the open-source Stanza toolkit.

Before you watch

• Assignment 2 in this course pairs directly with this lecture, since it asks you to implement the machine-learning decision step of a neural dependency parser.
• Familiarity with basic parts of speech (noun, verb, adjective, preposition) is assumed; a prior linguistics class is not required.

Check your understanding

1. What is the key structural difference between constituency grammar and dependency grammar?
2. Why is human language described as "globally ambiguous" while programming language syntax is not?
3. What role did annotated treebanks play in moving NLP away from hand-written grammar rules?
4. In transition-based parsing, what do the shift, left-arc, and right-arc operations each do to the stack and buffer?
5. What made the neural transition-based parser both faster and more accurate than earlier symbolic feature-based parsers?