Self-supervised learning

As explained in The AI Pocket Book

A training approach where a model learns from automatically generated targets derived from the input data itself.

Self-supervised learning generates labeled examples automatically from existing data. It remains supervised learning because the model learns from input–target pairs; “self” means the targets come from the data rather than manual labeling. section 1.27

Guessing the missing token

Take “The Eiffel Tower is in Paris,” remove the final token, and train the model to predict “Paris.” Repeating this across huge text collections creates next-token training examples for a language model. section 1.27

The basic recipe is straightforward:

• Collect a large amount of existing data, such as internet text. section 1.27 section 1.28
• Corrupt or subdivide each input so part of the original data becomes the target. section 1.27
• Train the model to reconstruct the missing information, commonly by guessing the next token. section 1.27
• Treat scale as an advantage, but not a guarantee: self-supervised models can perform impressively while still making serious mistakes. section 1.28

The method works when the task can be framed as reconstructing corrupted input. Early large language models used this approach with public text, then received manually generated data to align their behavior with user objectives. section 1.27 section 1.34

Figure 1.10 Training examples are generated by subdividing existing sentences and turning the last token in each into the desired autocomplete label.

Visualizes how subdividing sentences turns the final token into an autocomplete label. section 1.27

How The AI Pocket Book covers it

How deep it goes

Explains automatic labeling, input corruption, next-token reconstruction, and the boundary between self-supervised and supervised learning.

Examples

Uses an Eiffel Tower sentence, next-token prediction, Common Crawl, and Books3 as concrete training examples or data sources.

What you can do after

Evaluate how automatically generated reconstruction targets can scale language-model training while recognizing that performance remains imperfect.

How it connects

Connects supervised learning, unsupervised learning, large language models, and manual alignment data.

liveBookmanning.com

Read The AI Pocket Book on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionSelf-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
• 

  liveBookmanning.com
  Outlier Detection in PythonSelf-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA training setup that derives supervisory signals from the data itself rather than from human-provided labels.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionSelf-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
• 

  liveBookmanning.com
  AI-Powered SearchSelf-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.
• 

  liveBookmanning.com
  Generative AI in ActionA machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.

Compare how 8 books explain self-supervised learning →

Related concepts

• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
• pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
• unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
• foundational modelused by · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
• trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of The AI Pocket Book. The section links point to the parts of the book it draws on.