liveBook · Manning

RedPajama

As explained in LLMs in Production

RedPajama is a large open text dataset designed to reproduce the data composition described for the LLaMA model.

RedPajama is a fully open text dataset designed to reproduce the data composition described for LLaMA. It targets large-scale language-model training rather than routine fine-tuning. section 4.3.1 section 4.1.3

Its practical profile is defined by scale, breadth, and a limited number of very large sources:

  • Combines GitHub, arXiv, Books, Wikipedia, StackExchange, multiple Common Crawl dumps, and standard C4. section 4.3.1
  • Reaches about 5 TB and was published in April 2023, making it newer and larger than The Pile. section 4.3.1
  • Is distributed through the Hugging Face Hub as an openly available resource. section 4.3.1
  • Usually matters most when training a language model from scratch; fine-tuning projects are unlikely to need the full dataset. section 4.3.1

A LLaMA-oriented data mixture

A pretraining pipeline can treat RedPajama as a broad mixture of code, scholarly writing, books, reference text, discussion pages, and filtered web snapshots rather than as one homogeneous corpus. section 4.3.1 section 4.1.3

Table 4.2 Summary of datasets
Dataset
Contents
Size
Last update
Wikitext
English Wikipedia
<1 GB
2016
Wiki-40B
Multi-lingual Wikipedia
10 GB
2020
Europarl
European Parliament proceedings
1.5 GB
2011
Common Crawl
The internet
~300 GB
Ongoing
OpenWebText
Curated internet using Reddit
55 GB
2019
The Pile
Everything above plus specialty datasets (books, law, med)
825 GB
2020
RedPajama
GitHub, arXiv, Books, Wikipedia, StackExchange, and multiple version of Common Crawl
5 TB
2023
OSCAR
Highly curated multilingual dataset with 166 languages
9.4 TB
Ongoing

Use the dataset comparison table to place RedPajama’s 5 TB scale and source coverage beside The Pile, Common Crawl, and OSCAR. section 4.3.1

How LLMs in Production covers it

How deep it goes
Defines purpose, components, scale, date, and access, but does not detail preprocessing or filtering procedures.
Examples
Uses GitHub, arXiv, Books, Wikipedia, StackExchange, Common Crawl dumps, and C4 as constituent sources.
What you can do after
Evaluate whether an open, LLaMA-oriented corpus fits large-scale pretraining rather than ordinary fine-tuning.
How it connects
Connects LLaMA, The Pile, Common Crawl, C4, Hugging Face Hub, and language-model pretraining.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.