Lecture 16: ConvNets and Tree Recursive Neural Networks

What this lecture covers

This lecture steps outside the Transformer-dominated mainstream to cover two older neural architectures for language: convolutional neural networks (CNNs) and tree recursive neural networks. Both were influential ideas that are rarely used today, but the lecture presents them because understanding earlier approaches shows why certain design choices, such as max pooling, residual connections, and tree-structured composition, were tried and later reappeared or were abandoned.

After watching, you should be able to explain how n-gram convolutions and pooling work for text, describe two published CNN architectures for text classification, and explain how tree recursive neural networks compute sentence meaning from a parse tree, including why they struggled with negation compared to later models.

Key ideas

• N-gram convolution: a filter slides over a sequence of word vectors, computing a value for each n-gram (bigram, trigram, etc.), analogous to a 2D convolution sliding over image patches.
• Max pooling: keeps the strongest activation of each filter across the whole text, acting like a feature detector that fires if a pattern appears anywhere in the input; it generally outperforms average pooling for this kind of feature.
• Padding, stride, and dilation: padding preserves sequence length after convolution, stride controls how much filters overlap, and dilation lets a filter see spaced-out words rather than only adjacent ones.
• Yoon Kim's sentence classifier (2014): a simple CNN using multiple filter sizes, max pooling, and two copies of word vectors (fine-tuned and frozen) that matched or beat prior sentiment classifiers.
• Very deep character-level CNN (Conneau et al., 2017): borrows vision-style deep architectures (with residual connections and pooling) applied directly to characters, reaching competitive text classification results without pretrained word vectors.
• Tree recursive neural networks: compute a phrase's representation from its child representations following the sentence's parse tree, motivated by the principle of compositionality.
• Recursive Neural Tensor Network (RNTN): extends the basic tree model with a multiplicative (tensor) interaction between child vectors, letting composition depend on which words are combined.
• Negation handling: the RNTN, trained on the Stanford Sentiment Treebank, correctly modeled cases like a double negative flipping a negative phrase back to positive, something bag-of-words and earlier tree models could not do.

Walkthrough

N-gram convolutions over word vectors (4:07)

After brief course-logistics remarks, the lecture introduces CNNs for language by contrasting them with recurrent networks, which build a representation only by reading left to right. A convolution instead treats a sentence as a 1D signal: a filter is applied to each n-gram of word vectors (for example, every trigram), computing a dot product plus a bias term passed through a nonlinearity. This is presented as the language analogue of the 2D convolution used in vision, where a filter slides over image patches to detect visual patterns regardless of position.

Pooling, padding, stride, and dilation (10:13)

With multiple filters producing multiple values per n-gram position, the lecture explains how to summarize them. Max pooling takes the strongest activation for each filter across the whole sequence, functioning like a feature detector that reports whether a pattern (such as first-person language) occurs anywhere in the text. Average pooling is offered as an alternative for measuring a continuous quality of the text, but max pooling tends to work better in practice. The lecture also covers padding (to keep output length equal to input length), stride (skipping positions to reduce overlap between n-grams), local max pooling, k-max pooling (keeping the top k activations), and dilation (combining non-adjacent words in a filter), noting these last techniques see more use in speech than NLP.

Yoon Kim's CNN sentence classifier (18:21)

The lecture covers a widely cited 2014 paper that applied CNNs to sentiment classification. The model runs several filter sizes (bigram, trigram, four-gram) over a sentence, max-pools each filter's output, concatenates the results into one vector, and feeds it to a softmax classifier. A key detail is the "fine-tuned vs. frozen" trick: because fine-tuning pretrained word vectors on a small supervised dataset can distort the meaning of words absent from that dataset (the lecture illustrates this with "tedious," "dull," and "plotting" drifting apart), the model keeps two channels per filter, one using fine-tuned vectors and one using the original vectors. This simple architecture matched or beat many contemporary models across several sentiment and classification datasets, although the comparison is complicated by the fact that Dropout, which the CNN used, had not yet been adopted by the older baselines.

A very deep, character-level CNN (32:33)

The lecture then covers a 2017 architecture (Conneau et al.) that borrows the deep, vision-style design used in models like ResNet, in contrast to the shallow (2-4 layer) sequence models common in NLP at the time. Instead of starting from pretrained word vectors, it processes raw characters through stacked convolutional blocks with residual connections and periodic pooling that halves sequence length while doubling filter count, followed by k-max pooling and fully connected layers, closely mirroring vision architectures like VGGNet. Tested at depths of 9, 17, and 29 layers on text classification datasets (news categorization, sentiment on Yelp and Amazon reviews), the deepest 29-layer model performed best overall and matched or beat the previous best published results on several datasets, showing that character-level, vision-style CNNs could reach competitive results without any pretrained word representations.

Tree recursive neural networks and compositionality (42:42)

Shifting to a different family of models, the lecture introduces tree recursive neural networks, developed largely at Stanford, motivated by the recursive, nested structure of human language (illustrated with deeply embedded noun phrases and a Penn Treebank parse full of nested verb phrases). The core idea follows the principle of compositionality: a phrase's meaning is computed from its children's word vectors combined through a shared neural network, producing both a parent vector and a score for how plausible that combination is as a constituent. A greedy parser can use these scores to build up a binary parse tree from the words up, producing a vector representation for the whole sentence. The lecture notes this basic version, using a single shared weight matrix at every node, had a real limitation: it could not represent that different word combinations (like adjective-noun versus verb-object) interact in different ways.

The Recursive Neural Tensor Network and sentiment with negation (54:59)

To address that limitation, the lecture presents the Recursive Neural Tensor Network (RNTN), which replaces the simple linear combination of child vectors with a multiplicative tensor interaction, allowing composition to depend on which words or phrases are being combined. This model was trained on the Stanford Sentiment Treebank, a dataset the lecture describes as sentiment-labeling every phrase in nearly 12,000 parsed sentences, not just whole sentences. Training on every tree node (rather than only sentence-level labels) improved even a simple bigram naive Bayes baseline from 79% to 83% accuracy, and the RNTN itself outperformed bigram naive Bayes further. The most notable results were qualitative: the RNTN correctly handled cases such as "it's just incredibly dull" staying negative despite "incredible" being individually positive, and correctly flipped a doubly negated negative phrase ("definitely not dull") back to positive, a case that neither naive Bayes nor earlier tree models handled correctly. The lecture closes by noting that tree recursive networks lost out to Transformers because their tree-structured information flow is more restrictive than full attention, even though they may better capture certain compositional phenomena like negation.

Before you watch

• Familiarity with word vectors (GloVe or word2vec) and basic feedforward neural network layers is assumed throughout.
• It helps to have seen the course's earlier lecture on recurrent neural networks, since CNNs are introduced partly by contrast with them.
• Some background in constituency parsing or phrase-structure trees makes the tree recursive network section easier to follow.

Check your understanding

1. How does max pooling act like a feature detector when applied to the output of a text convolution filter?
2. Why did Yoon Kim's model keep two copies of the word vectors, one fine-tuned and one frozen?
3. What design choices does the deep character-level CNN borrow from vision architectures, and what does it deliberately avoid using?
4. How does a tree recursive neural network decide which pairs of words or phrases to combine when building a parse tree?
5. Why could the Recursive Neural Tensor Network correctly interpret a doubly negated phrase like "definitely not dull" when earlier models could not?