Self-supervised learning
Self-supervised learning is a machine-learning approach that derives training targets or supervisory signals from the data itself rather than relying on externally supplied labels.
Build AI Drug Discovery Pipelines
Self-supervised learning trains a model using supervisory signals or prediction targets derived from the input data itself rather than external labels.
- How it is used
- Learns representations from unlabeled protein sequences and cellular images for downstream biological predictions.
- How deep it goes
- Works through masked protein-sequence prediction, masking proportions, and evaluation; briefly describes image-based objectives.
- Examples
- Uses a masked protein sequence, ESM-2 protein embeddings, and cellular images of genetic and chemical perturbations.
- What you can do after
- Build a small protein language model from scratch or apply pretrained ESM-2 to extract protein embeddings.
Read how Build AI Drug Discovery Pipelines explains self-supervised learning →
LLMs in Production
Self-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
- How it is used
- Uses self-supervised prediction to pretrain language models on text without manual labels.
- How deep it goes
- Outlines prediction tasks and data-preparation considerations, while deferring detailed pipeline construction.
- Examples
- Identifies C4 as a common pretraining dataset and notes its multilingual content, code, and societal bias.
- What you can do after
- Evaluate/understand self-supervised pretraining data needs and the continuing importance of cleaning and bias checks.
Read how LLMs in Production explains self-supervised learning →
Outlier Detection in Python
Self-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
- How it is used
- Creates training targets from unlabeled data to learn representations for deep learning-based outlier detection.
- How deep it goes
- Explains a rotation-prediction pretext task and its embeddings, then identifies tabular approaches without specifying a complete tabular detection procedure.
- Examples
- Uses four rotations of 100,000 images and considers arithmetic changes and shuffled cells as tabular transformations.
- What you can do after
- Evaluate/understand when automatically generated training targets might yield useful outlier-detection embeddings, especially why tabular data poses difficulties.
Read how Outlier Detection in Python explains self-supervised learning →
The AI Pocket Book
A training approach where a model learns from automatically generated targets derived from the input data itself.
- How it is used
- Generates vast numbers of training examples for language models by extracting targets automatically from existing data.
- How deep it goes
- Explains automatic labeling, input corruption, next-token reconstruction, and the boundary between self-supervised and supervised learning.
- Examples
- Uses an Eiffel Tower sentence, next-token prediction, Common Crawl, and Books3 as concrete training examples or data sources.
- What you can do after
- Evaluate how automatically generated reconstruction targets can scale language-model training while recognizing that performance remains imperfect.
Read how The AI Pocket Book explains self-supervised learning →
Inside Deep Learning: Math, Algorithms, Models
A training setup that derives supervisory signals from the data itself rather than from human-provided labels.
- How it is used
- Reduces dependence on expensive manual labeling while training models on abundant unlabeled data.
- How deep it goes
- Explains implicit targets, autoencoding constraints, sequence losses, and sampling, but defers broader topic-modeling methods.
- Examples
- Uses masked images, shuffled image pieces, reconstructed inputs, and character-level Shakespeare generation from a seed.
- What you can do after
- Apply it via PyTorch by constructing input-derived targets, training with standard losses, and sampling autoregressive outputs.
Read how Inside Deep Learning: Math, Algorithms, Models explains self-supervised learning →
Build a Large Language Model (From Scratch)
A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
- How it is used
- Uses input text to create supervision for LLM pretraining before supervised fine-tuning.
- How deep it goes
- Explains on-the-fly next-word labels and GPT’s autoregressive flow, but defers implementation details.
- Examples
- Uses raw text, sentence prefixes, upcoming words, and GPT text completion as concrete training examples.
- What you can do after
- Apply self-supervision by training a GPT-style model to predict upcoming words from large unlabeled text datasets.
Read how Build a Large Language Model (From Scratch) explains self-supervised learning →
Timeless Algorithms, The Foundational Ideas
Self-supervised learning trains a model using supervisory signals derived from the data itself rather than manually supplied labels.
- How it is used
- Reduces dependence on manually supplied labels by deriving prediction targets from the data itself.
- How deep it goes
- Defines data-derived targets and gives prediction examples without detailing model internals or optimization.
- Examples
- Uses missing-image prediction, next-element prediction, and prediction of concealed content as task examples.
- What you can do after
- Understand how data-derived prediction targets can reduce reliance on manually labeled training data.
Read how Timeless Algorithms, The Foundational Ideas explains self-supervised learning →
Deep Learning with R, Third Edition
Self-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
- How it is used
- Trains foundation models on massive unlabeled datasets before adapting their learned representations to downstream applications.
- How deep it goes
- Defines input-derived targets and reconstruction objectives, but stops short of detailing optimization or implementation procedures.
- Examples
- Uses next-word prediction, noisy-image reconstruction, and partial-input reconstruction as concrete training examples.
- What you can do after
- Evaluate/understand how unlabeled data can train reusable representations for varied downstream tasks.
Read how Deep Learning with R, Third Edition explains self-supervised learning →
AI-Powered Search
Self-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.
- How it is used
- Pretrains language models and embeddings from large unlabeled text collections before downstream search or language tasks.
- How deep it goes
- Explains token masking, contextual prediction, loss optimization, and embedding formation without detailing complete Transformer internals.
- Examples
- Uses randomly masked sentence tokens and the Transformer encoder as concrete training examples.
- What you can do after
- Evaluate/understand how unlabeled text becomes contextual embeddings through masked-token prediction.
Read how AI-Powered Search explains self-supervised learning →
Generative AI in Action
A machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.
- How it is used
- Supports foundational models that generalize across tasks and power generative AI applications.
- How deep it goes
- Defines the technique and its role in representation learning but does not detail training objectives or implementation steps.
- Examples
- Uses translation, question answering, object recognition, scene understanding, and robotic navigation as application examples.
- What you can do after
- Evaluate/understand how models learn reusable representations from unlabeled multimodal data.
Read how Generative AI in Action explains self-supervised learning →
Also covered in
- Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
- Deep Learning with PyTorch, Second Edition
- Natural Language Processing in Action, Second Edition
- Introduction to Generative AI, Second Edition
- Deep Learning with Python, Third Edition
- Self-Improving AgentsIn early access (MEAP)
- Retrieval Augmented Generation, The Foundational IdeasIn early access (MEAP)
- Domain-Specific Small Language Models
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
- generationGeneration is the production of an output by a computational model from supplied input or context, often using patterns learned from data.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- language modelused by · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
- pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
- unlabeled datauses · Unlabeled data is a collection of examples that lack target labels or task-specific annotations.
- foundation modelused by · A foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
Questions about self-supervised learning
What is self-supervised learning?
Self-supervised learning is a machine-learning approach that derives training targets or supervisory signals from the data itself rather than relying on externally supplied labels.
Which Manning books explain self-supervised learning?
Build AI Drug Discovery Pipelines, LLMs in Production, Outlier Detection in Python, The AI Pocket Book, Inside Deep Learning: Math, Algorithms, Models, Build a Large Language Model (From Scratch), Timeless Algorithms, The Foundational Ideas, Deep Learning with R, Third Edition and 2 more.
What should I know before learning self-supervised learning?
Manning books that teach self-supervised learning usually introduce language model, natural language processing, generation and transformer architecture first.
What is self-supervised learning a kind of?
Self-supervised learning is a kind of machine learning. Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
What is self-supervised learning also called?
Self-supervised learning is also called self-supervision.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.









