How deeply does LLMs in Production cover self-supervised learning?

Explains prediction targets and outlines data-preparation steps, but defers detailed pipeline construction and model internals.

Self-supervised learning

As explained in LLMs in Production

Self-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.

Suppose your training data is a giant text dump with no labels or annotations. For an LLM, that does not prevent training: self-supervised methods use the text to set prediction tasks, such as predicting the next word or a masked word. This is why many traditional labeling and feature-engineering steps are unnecessary for pretraining. section 4.3.2

No manual labels does not mean no data preparation. A practical workflow starts by defining a schema, checking whether feature distributions suit the intended outcome, looking for bias and anomalies, and converting text into model-ready form through tokenization and embedding. These are high-level considerations, not a complete data-pipeline recipe. section 4.3.2

Corpus choice brings trade-offs:

C4 is a commonly used cleaned Common Crawl dataset for self-supervised pretraining, but cleaning does not remove inherent societal bias. section 4.3.1
Common Crawl offers diverse web text, languages, and code, alongside the internet’s less desirable content; its available variants differ substantially in size and filtering. section 4.3.1
OpenWebText is a smaller, 55 GB pretraining option assembled from URLs found through Reddit and filtered using karma rankings. section 4.3.1

More text alone is not a quality strategy. Large datasets can support convincing syntax without establishing semantics or pragmatics. Treat curation and informed bias checks as part of the work, even when the training targets come from the text itself. section 4.3.2

Table 4.2 Summary of datasets

Dataset	Contents	Size	Last update
Wikitext	English Wikipedia	<1 GB	2016
Wiki-40B	Multi-lingual Wikipedia	10 GB	2020
Europarl	European Parliament proceedings	1.5 GB	2011
Common Crawl	The internet	~300 GB	Ongoing
OpenWebText	Curated internet using Reddit	55 GB	2019
The Pile	Everything above plus specialty datasets (books, law, med)	825 GB	2020
RedPajama	GitHub, arXiv, Books, Wikipedia, StackExchange, and multiple version of Common Crawl	5 TB	2023
OSCAR	Highly curated multilingual dataset with 166 languages	9.4 TB	Ongoing

Compare candidate corpora by contents, size, and update date before choosing data for pretraining. section 4.3.1

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Outlier Detection in PythonSelf-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
liveBook manning.com

The AI Pocket BookA training approach where a model learns from automatically generated targets derived from the input data itself.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA training setup that derives supervisory signals from the data itself rather than from human-provided labels.
liveBook manning.com

Build a Large Language Model (From Scratch)A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
liveBook manning.com

Deep Learning with R, Third EditionSelf-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
liveBook manning.com

AI-Powered SearchSelf-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.
liveBook manning.com

Generative AI in ActionA machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.

Compare how 8 books explain self-supervised learning →

Related concepts

language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
foundational modelused by · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.