Lecture 14: Data Filtering and Deduplication

What this lecture covers

The previous lecture surveyed which datasets language models are trained on. This one goes underneath that survey to the mechanics: given a small set of high-quality target data and a huge pool of raw data such as Common Crawl, how do you build a fast, scalable algorithm that selects the subset of the raw data that resembles the target? The lecture works through three filtering primitives, shows how the same primitive is reused for language identification, quality filtering, and toxicity filtering, and then turns to deduplication, a separate but related problem of removing redundant text.

After watching, you should be able to explain how n-gram perplexity, linear classifiers, and importance resampling are each used to score documents, understand why deduplication needs pairwise comparisons, and describe how Bloom filters and MinHash locality-sensitive hashing make both exact and approximate deduplication run in roughly linear time over web-scale data.

Key ideas

• Filtering as generalization: every filtering method fits a model on a small target dataset and a large raw dataset, then scores and keeps raw documents that resemble the target; the method must be fast enough to run over the entire web.
• N-gram perplexity filtering: an n-gram language model (such as one trained with Kneser-Ney smoothing via the KenLM toolkit) can score documents by perplexity; CCNet used this to keep the lowest-perplexity third of paragraphs relative to Wikipedia.
• FastText classifiers: a near-linear classifier over hashed word or n-gram features that is fast enough to run over billions of documents, used for quality, language, and toxicity filtering.
• Importance resampling: fits generative distributions to both the target and raw data and resamples raw documents in proportion to the ratio of target to raw probability, aiming to match the target distribution rather than just classify membership in it.
• Model-generated targets: instead of only using existing curated sources as the target set, a strong language model (like GPT-4 in the Phi paper) can be prompted to generate or label the target examples used to train a fast classifier.
• Deduplication units and matching: deduplication requires choosing a unit (sentence, paragraph, document), a matching rule (exact or similarity-based), and an action (remove all but one, or all).
• Bloom filters: a memory-efficient probabilistic set-membership structure using multiple hash functions, used for fast, approximate exact-match deduplication with a tunable false-positive rate.
• MinHash and locality-sensitive hashing (LSH): a hashing scheme where the probability of collision approximates Jaccard similarity, combined into bands of hash functions to sharply separate near-duplicate pairs from dissimilar ones in roughly linear time.

Walkthrough

Filtering as a target-vs-raw problem (0:05)

The lecture reframes the previous lecture's filtering steps as one abstract problem: given a small target dataset representing what you want and a large raw dataset, find a subset of the raw data that resembles the target. Any solution needs to generalize beyond the exact target examples and needs to be cheap enough to run over the entire raw pool, since using an expensive model for filtering could cost as much as training itself.

N-gram models for scoring documents (1:06)

The instructor reviews how n-gram language models estimate conditional word probabilities by counting n-grams and applying Kneser-Ney smoothing to handle unseen combinations, using the KenLM library as the common implementation. A model trained on Wikipedia is used to score example sentences by perplexity: well-formed English gets lower perplexity than gibberish, though the method can be fooled since it only captures local word patterns. This is the technique CCNet used to keep the lowest-perplexity third of Common Crawl paragraphs when building the first Llama dataset.

FastText and linear classification (8:14)

FastText is introduced as a near-linear text classifier that reduces a huge vocabulary-sized weight matrix to a smaller hidden dimension, making it fast enough for web-scale filtering, and that extends to n-grams by hashing them into a fixed number of bins to bound the parameter count. The lecture notes the fundamental trade-off: bigger models like BERT could classify more accurately, but the compute spent filtering has to stay a small fraction of the compute spent training, since so much of the raw data will be discarded.

Importance resampling (13:16)

Building on the idea of importance sampling from Monte Carlo methods, this section shows how to fit simple hashed n-gram distributions to both the target and raw datasets, then resample raw documents proportional to the ratio of target to raw probability. Unlike a classifier that only decides membership, this method tries to match the shape of the target distribution, which can help preserve diversity, though the lecture notes the gains over fastText-style classification are modest in practice.

Applying filtering: language, quality, and toxicity (23:22)

The same scoring machinery is shown solving different problems. Language identification uses an off-the-shelf fastText classifier (used by Dolma to keep pages with English probability above 0.5), with a live demo showing it struggles on short or mixed-language text. Quality filtering is illustrated through GPT-3's classifier (curated sources vs. Common Crawl), Llama's classifier (pages referenced by Wikipedia vs. Common Crawl), and the Phi paper's approach of prompting GPT-4 to label a small set of Python code by educational value and using those labels to train a classifier over a much larger pool. Toxicity filtering follows the same pattern using the Jigsaw Toxic Comments dataset to train hate-speech and NSFW classifiers, as done in Dolma.

Deduplication: exact matches and Bloom filters (36:35)

Deduplication is presented as a distinct, pairwise problem: exact duplicates arise from mirrored sites, and near-duplicates arise from boilerplate text like license notices or templated content with only small variations, sometimes repeated tens of thousands of times in a corpus. Because comparing every pair of documents is too slow at web scale, the lecture introduces hash functions as the building block and demonstrates exact deduplication with Bloom filters: a compact bit array populated by multiple hash functions per item, which can test set membership with no false negatives and a tunable, small false-positive rate, as used in Dolma's paragraph-level deduplication.

Near-duplicate detection with MinHash and LSH (58:11)

To catch near-duplicates, the lecture defines Jaccard similarity between two sets and introduces MinHash, a hash function whose collision probability equals the Jaccard similarity of the two sets being compared, turning a pairwise computation into something approximable with independent hashing. Locality-sensitive hashing then groups multiple MinHash functions into bands, so that two documents collide if any band matches entirely; tuning the number of bands and hash functions per band sharpens the similarity threshold, letting pipelines like the one in a paper cited from the prior lecture set a near-duplicate threshold around 0.99 Jaccard similarity.

Before you watch

• Watch the previous lecture in this course (Lecture 13: Data) first, since it introduces the datasets and filtering steps this lecture explains algorithmically.
• Basic familiarity with hash functions, probability, and n-gram language models will make the deduplication and importance-resampling sections easier to follow.

Check your understanding

1. Why must a filtering algorithm be both fast and generalize beyond the exact target dataset it is given?
2. How does importance resampling differ from a fastText classifier in what it tries to achieve?
3. Where did the target dataset come from in the Phi paper's quality-filtering approach, and why did that reduce cost?
4. Why can't deduplication be solved with a document-by-document classifier the way quality filtering can?
5. How does increasing the number of bands versus the number of hash functions per band change a MinHash LSH deduplication threshold?