Lecture 13: Data

What this lecture covers

Earlier lectures assumed a fixed training set and focused on architecture, optimization, tokenization, scaling laws, and parallelism. This lecture turns to the question those lectures avoided: what data actually goes into a language model, and why model developers say so little about it. The instructor argues that data, more than architecture, is what differentiates one model from another, and traces the history of pre-training corpora from BERT through recent open releases like DCLM and Nemotron-CC.

After watching, you should be able to describe the pipeline that turns a raw web crawl into training tokens, name the major open pre-training datasets and how each one filters Common Crawl, and explain the legal basis (and limits) of training on copyrighted web content. The lecture also outlines how mid-training and post-training data differ from pre-training data, setting up later lectures on alignment.

Key ideas

• Training stages: pre-training uses large amounts of raw web data, mid-training curates smaller high-quality sets for specific capabilities (math, code, long context), and post-training fine-tunes on instruction and chat data.
• Common Crawl: a monthly, publicly available web crawl since 2007 that most open pre-training datasets start from; it is large but noisy, biased toward text, and not comprehensive.
• Model-based vs. rule-based filtering: some pipelines (CCNet, DCLM, Nemotron-CC) train classifiers to find pages that resemble Wikipedia or instruction data, while others (C4, Gopher, RefinedWeb) rely on hand-written heuristics to avoid biasing the data.
• Wikipedia as a quality surrogate: because Wikipedia is well-curated, many filtering pipelines use it (or pages it links to) as a stand-in for what "high quality" text looks like, even though this misses whole categories of useful content.
• Copyright and fair use: almost everything on the web is copyrighted by default; training on it without a license depends on a fair-use argument, and the situation is legally unsettled.
• Shadow libraries: datasets like Books3 and LibGen supply large amounts of book text but were built by disregarding copyright, and their use has led to lawsuits.
• Instruction data: post-training datasets (Alpaca, Vicuna, OpenHermes, Tulu) are increasingly built by prompting existing language models rather than collecting data from scratch.

Walkthrough

Why data is the differentiator (0:05)

The lecture opens by noting that model papers like Llama 3's disclose architecture and training details in depth but say very little about the data itself, largely for competitive and legal reasons. The instructor argues data deserves the most attention because it is highly parallelizable across a team (different people can work on code, multilinguality, math, and so on) in a way architecture work is not. The lecture then defines the three stages of training: pre-training on raw web data, mid-training on curated high-quality subsets aimed at specific capabilities, and post-training on instruction and chat data, noting that in practice these stages blur together.

From BERT to Common Crawl (6:14)

Starting from BERT (2018), which trained on books and Wikipedia, the lecture walks through early corpora: the Smashwords-derived BooksCorpus, and Wikipedia itself, including a digression on data poisoning, where an attacker can time malicious edits to land in a periodic Wikipedia dump. GPT-2's WebText dataset is introduced as an early attempt to filter the web cheaply, by keeping only pages linked from Reddit posts with enough upvotes. This leads into Common Crawl: how it crawls the web using seed URLs and a crawl frontier, respects (or is ignored by) robots.txt, and produces both raw HTML (WARC) and extracted text (WET) formats, where the choice of HTML-to-text tool can measurably affect downstream quality.

Filtering Common Crawl: CCNet, C4, and the Pile (21:33)

The lecture compares two early filtering approaches: CCNet, which trains an n-gram model on Wikipedia to score how "Wikipedia-like" a document is, and C4, which uses purely rule-based heuristics (removing short pages, bad words, and code-like text). It then covers GPT-3's dataset, which combines Common Crawl with WebText2, books, and Wikipedia and uses a quality classifier trained to distinguish curated sources from the rest. The Pile, assembled by EleutherAI, is presented as a more deliberately diverse, community-curated mix of 22 domains including PubMed Central, arXiv, Enron emails, Project Gutenberg, Books3, Stack Exchange, and GitHub, each with its own processing quirks.

Newer pre-training datasets (42:51)

The lecture surveys Gopher's MassiveText (rule-based filtering), Llama's Common Crawl mix filtered with a classifier trained on pages linked from Wikipedia, and RedPajama as an open reproduction. RefinedWeb and FineWeb are introduced as datasets that argue heavy filtering of web text alone, without mixing in curated sources, can be enough. Dolma, from AI2, and DataComp's DCLM baseline (which aggressively filters Common Crawl using a classifier trained on instruction-like data) are covered, along with Nvidia's Nemotron-CC, which pushes token yield higher by ensembling multiple quality classifiers and using a language model to rewrite low-quality pages and generate synthetic instruction-style pairs from high-quality ones.

Copyright and fair use (1:00:10)

This section explains that copyright applies automatically to original creative works without requiring registration, and that most of the web is technically copyrighted. Training on such data legally requires either a license (contracts, or Creative Commons licenses like those covering much of Wikipedia) or an appeal to fair use, which depends on factors like the purpose of use, the nature of the original work, and effect on the market for that work. The lecture notes that copyright covers more than verbatim copying, plot and characters can also be protected, and that even licensed or fair-use content may still be off-limits under a platform's terms of service.

Mid-training and post-training data (1:09:27)

The final section covers long-context extension, typically added at mid-training using books and math data with long-range dependencies, and the evolution of instruction-tuning datasets: early NLP-task collections like Super-Natural Instructions and FLAN, synthetic approaches like Alpaca's self-instruct and Vicuna's ShareGPT conversations, and more recent aggregations like OpenHermes and the Llama-Nemotron post-training data. The lecture closes by noting that generating instruction data by distilling open-weight models is more permissive than distilling GPT-4, and that hiring human annotators remains the most controlled but expensive option.

Before you watch

• Review earlier lectures on tokenization and scaling laws, since this lecture assumes you already know how a model consumes tokens and why data quantity interacts with model size.
• Some familiarity with how web crawlers work (URLs, HTML, robots.txt) will make the Common Crawl discussion easier to follow.

Check your understanding

1. Why does the instructor argue that data is more important than architecture for differentiating language models?
2. How do CCNet and C4 differ in their approach to filtering Common Crawl?
3. What is the difference between how DCLM baseline and RefinedWeb decide whether to use model-based filtering?
4. Under what conditions might using copyrighted web text for training be considered fair use?
5. Why do many recent instruction-tuning datasets rely on generating data from existing language models rather than manual collection?