Lecture 16: Multimodal Deep Learning, with Douwe Kiela

What this lecture covers

Douwe Kiela, a researcher who has worked on multimodal models at Facebook AI Research and Hugging Face, gives a guest lecture surveying how deep learning has combined text with other modalities, mainly images. The lecture answers a practical question for anyone building or evaluating multimodal systems: what are the main architectural approaches to combining information from different modalities, and how has the field progressed from simple embedding alignment to today's large foundation models. It fits into the course as a look beyond pure text, showing how techniques already covered (word embeddings, attention, Transformers) extend to other data types.

After watching, you should be able to describe why multimodal training matters, distinguish early, middle, and late fusion strategies, explain how contrastive models like CLIP work, and recognize common pitfalls in multimodal evaluation datasets.

Key ideas

• Modality: a type of information source, such as text, image, audio, or smell; multimodal models combine several of these.
• Motivations for multimodality: it mirrors how humans perceive the world, matches the multimodal nature of real internet content, and offers a way to keep scaling models as high-quality text data runs out.
• Fusion: the general problem of how to combine feature vectors from different modalities, ranging from early fusion (interaction from the start) to late fusion (combining only final scores).
• Contrastive (late-fusion) models: models like CLIP encode each modality independently and align matching pairs in a shared embedding space using a contrastive loss, enabling efficient training and zero-shot classification.
• BERT-style multimodal transformers: models like VisualBERT and ViLBERT extend masked-language-model pretraining to combine region-based image features with text, using single-stream or dual-stream (cross-attention) architectures.
• Foundation models: more recent systems (FLAVA, Flamingo, BLIP-2) aim for one model that handles vision, language, and multimodal tasks together, often by freezing a pretrained language model and learning to project image features into its input space.
• Evaluation pitfalls: several popular benchmarks (early VQA, some multimodal pretraining gains) turned out to be solvable without real multimodal reasoning, motivating harder, more carefully designed datasets like Winoground and Hateful Memes.

Walkthrough

Why multimodality matters now (1:06)

Kiela opens by noting that "multimodality" is loosely defined, then gives three reasons it matters: human understanding of the world is inherently multimodal, real internet content mixes text and images constantly, and the field is starting to run out of pure text data for scaling large language models, making other modalities a way to keep growing training data. He illustrates the perceptual point with the McGurk effect, where identical audio is perceived differently depending on the video shown alongside it, and lists common multimodal applications: retrieval, image captioning, text-to-image generation, visual question answering, and multimodal classification (such as hate speech detection).

Early cross-modal embeddings (7:10)

The lecture traces early pre-deep-learning and early deep learning work that aligned word embeddings with image features in a shared space using similarity metrics and margin losses, enabling cross-modal retrieval, such as finding images near the embedding for "horse." It covers the "bag of visual words" approach (clustering image keypoint descriptors the way bag-of-words clusters text), then how convolutional neural network features were concatenated with or used to predict word embeddings, followed by extensions from words to sentences using LSTMs and recursive networks, sequence-to-sequence captioning models, attention-based alignment between generated words and image regions, and GANs conditioned on text for image generation.

Featurizing images and fusion strategies (15:19)

Kiela explains why combining modalities is harder than it sounds: text can dominate a model's learned signal (some early visual question answering systems learned to ignore images entirely), extra modalities add noise, and data coverage is often incomplete. He describes ways to featurize images, from region features produced by object detectors to Vision Transformers that treat image patches like tokens, and surveys ways to combine two feature vectors, from simple similarity or concatenation to multiplicative interactions, gating, and attention. He frames most multimodal research as fundamentally about this fusion question, and distinguishes early fusion (interaction from the start), middle fusion, and late fusion (combining only final outputs), illustrated with the FiLM model, which modulates a ResNet's feature maps at every layer using a vector derived from the other modality.

Late fusion in practice: CLIP and contrastive pretraining (23:22)

The lecture covers CLIP as the most influential late-fusion, contrastive model: independent Transformer encoders for text and images are trained so that matching image-text pairs rank higher than mismatched pairs within a batch. Kiela credits CLIP's success less to architectural novelty than to training on roughly 300 million web-sourced image-text pairs, which produced a model that generalizes far better than a comparable ResNet to adversarial image variants, even though it didn't outperform ResNet on plain ImageNet. He mentions that CLIP-style captions (natural web captions rather than single labels) enable zero-shot classification through prompting, and describes follow-up work (Google's ALIGN, the open LAION datasets) that scaled the same idea to billions of pairs.

Early and middle fusion: BERT-style multimodal transformers (28:27)

Kiela walks through the wave of BERT-derived multimodal models that appeared once BERT and region-based image features were both available: VisualBERT as a simple single-stream model that concatenates image region features and text tokens into one Transformer, and ViLBERT as a dual-stream model with cross-attention between two parallel Transformers, trained with masked-language-modeling and image-text-matching objectives analogous to BERT's next-sentence prediction. He also mentions his own MMBT model, which simply pools ResNet feature maps into BERT's token space without extra multimodal pretraining, and Pixel-BERT and ViLT, which move toward using raw image patches instead of region features. He cites a paper showing that, once trained on identical data, many of these architecturally different models perform similarly, suggesting data and scale mattered more than specific fusion tricks.

Toward foundation models and generation (36:33)

The lecture covers FLAVA, Kiela's own project aiming for a single foundation model handling vision-only, language-only, and multimodal tasks together, combining masked image modeling, masked language modeling, image-text matching, and a CLIP-style contrastive loss. It then traces a shift toward generative multimodal models: SimVLM and CoCa add text decoders for richer captioning; frozen-language-model approaches (following the earlier MMBT idea) learn only a projection into a frozen model's token space, enabling few-shot, in-context "binding" behavior; and Flamingo (DeepMind) combines a frozen Chinchilla language model with a "Perceiver Resampler" and gated cross-attention to reason over multiple images. BLIP-2 pushes this further, freezing both the image encoder and the language model and learning only a small connecting module, producing strong captioning and visual question answering. Kiela closes this section by noting that multimodal chain-of-thought prompting, generating a rationale before answering, measurably improves performance on benchmarks requiring visual reasoning.

Evaluating multimodal models: pitfalls and better benchmarks (54:43)

Kiela argues that careful evaluation matters as much as model design, describing how early visual question answering (VQA) datasets could be gamed: guessing "two" for every counting question scored roughly 70% accuracy, revealing that images often weren't needed to answer. He describes datasets built specifically to prevent shortcuts: Hateful Memes, which he helped create, pairs benign and hateful versions of memes that differ only when image and text are read together, and found that fancy multimodal pretraining barely improved over simpler baselines; and Winoground, which tests whether models like CLIP truly understand word order and compositional relationships (such as "a mug in some grass" versus "grass in a mug"), where state-of-the-art models sometimes perform below random chance. He also notes that even a strong generative model like DALL-E 2 struggles with details like relative counts of objects, reflecting biases in its training data.

Before you watch

• Familiarity with word embeddings, attention, and the Transformer architecture from earlier lectures is assumed throughout.
• It helps to have seen the course's coverage of BERT and sequence-to-sequence models, since much of the lecture builds directly on both.

Check your understanding

1. What are the three main motivations Kiela gives for pursuing multimodal deep learning?
2. How does a contrastive model like CLIP differ from an early- or middle-fusion architecture like ViLBERT in how it combines modalities?
3. Why might comparing multimodal models trained on different datasets be misleading when judging architectural improvements?
4. What made the original VQA benchmark solvable without real multimodal reasoning, and how did Hateful Memes and Winoground try to avoid the same problem?
5. What role does freezing a pretrained language model play in architectures like the frozen-projection approach, Flamingo, and BLIP-2?