Self-supervised learning

As explained in AI-Powered Search

Self-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.

Self-supervised learning trains a model by deriving supervisory targets from the input data instead of requiring manually labeled examples. In language modeling, the model creates its own prediction task from ordinary text. section 13.3.1

A Transformer can randomly mask 15% of the tokens in a sentence and predict the missing words from surrounding context. This masked language modeling task, also called a Cloze test, optimizes a loss function; repeated training produces embeddings that capture words and their contexts. section 13.3.1 section 15.1.2

Masked-token pretraining

Given a sentence such as “The cat sat on the mat,” the training process can hide tokens and require the model to predict them using the remaining words. The resulting representations support later language and search tasks. section 13.3.1

Figure 13.4 The Transformer encoder

Trace the Transformer encoder pipeline used to turn tokenized text into contextual representations. section 13.3.1

How AI-Powered Search covers it

How deep it goes

Explains token masking, contextual prediction, loss optimization, and embedding formation without detailing complete Transformer internals.

Examples

Uses randomly masked sentence tokens and the Transformer encoder as concrete training examples.

What it assumes

Assumes basic familiarity with language models and embeddings while introducing masked prediction as the training objective.

What you can do after

Evaluate/understand how unlabeled text becomes contextual embeddings through masked-token prediction.

How it connects

Connects Transformers, BERT, masked language modeling, Cloze tests, embeddings, and fine-tuning.

liveBookmanning.com

Read AI-Powered Search on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionSelf-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
• 

  liveBookmanning.com
  Outlier Detection in PythonSelf-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
• 

  liveBookmanning.com
  The AI Pocket BookA training approach where a model learns from automatically generated targets derived from the input data itself.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA training setup that derives supervisory signals from the data itself rather than from human-provided labels.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionSelf-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
• 

  liveBookmanning.com
  Generative AI in ActionA machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.

Compare how 8 books explain self-supervised learning →

Related concepts

• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
• pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
• unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
• foundational modelused by · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
• trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.