Self-supervised learning

As explained in Inside Deep Learning: Math, Algorithms, Models

A training setup that derives supervisory signals from the data itself rather than from human-provided labels.

Self-supervised learning trains a neural network without human-provided labels by deriving targets from the input data itself. It still uses familiar regression or classification losses, but the prediction target is an implicit part of x. section 7.0 section 7.7

Common target-construction patterns include:

• Inpainting obscures part of an input and predicts the missing content. section 7.0
• Image sorting shuffles image pieces and predicts their original order. section 7.0
• Autoencoding predicts the original input, using a bottleneck or denoising transformation to prevent merely copying it. section 7.7
• Autoregressive modeling predicts the next sequence item, using x̂ᵢ against xᵢ₊₁ for each available timestep. section 7.5

Character-level generation

Treat a sentence as ordered characters. An autoregressive model predicts the next character from previous ones; after processing a seed such as “EMILIA:”, it samples each prediction as the next input and generates a new sequence. section 7.5.2

Figure 7.1 Three different types of self-supervised problems: inpainting, image sorting, and autoencoding. In each case, we do not need to know what the image is of because the network will try to predict the original image no matter what. In the first case (red), we randomly block out a portion of the image and ask the network to fill in the missing piece. In the second case, we break the image into pieces, shuffle them, and ask the network to put them back in the correct order. In the last case, we simply ask the network to predict the original image from the original image.

Compare three target-construction strategies—filling a masked image region, restoring shuffled pieces, and reconstructing the original image. section 7.0

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains implicit targets, autoencoding constraints, sequence losses, and sampling, but defers broader topic-modeling methods.

Examples

Uses masked images, shuffled image pieces, reconstructed inputs, and character-level Shakespeare generation from a seed.

What it assumes

Assumes familiarity with classification, regression, loss functions, and neural networks while teaching how to learn without labels.

What you can do after

Apply it via PyTorch by constructing input-derived targets, training with standard losses, and sampling autoregressive outputs.

How it connects

Connects unsupervised learning, regression, classification, autoencoding, autoregressive models, generative models, and transfer learning.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionSelf-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
• 

  liveBookmanning.com
  Outlier Detection in PythonSelf-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
• 

  liveBookmanning.com
  The AI Pocket BookA training approach where a model learns from automatically generated targets derived from the input data itself.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionSelf-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
• 

  liveBookmanning.com
  AI-Powered SearchSelf-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.
• 

  liveBookmanning.com
  Generative AI in ActionA machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.

Compare how 8 books explain self-supervised learning →

Related concepts

• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
• pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
• unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
• foundational modelused by · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
• trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.