Self-supervised learning
Self-supervised learning is a machine-learning approach that derives training targets or supervisory signals from the data itself rather than relying on externally supplied labels.
LLMs in Production
Self-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
- How it is used
- Enables LLM pretraining on text dumps without manually supplied labels by predicting the next or a masked word.
- How deep it goes
- Explains prediction targets and outlines data-preparation steps, but defers detailed pipeline construction and model internals.
- Examples
- Uses unlabeled text dumps, next-word and masked-word prediction, and corpora including C4 and OpenWebText.
- What you can do after
- Practice/perform it by selecting and preparing unlabeled text for LLM pretraining, while recognizing that cleaning remains necessary.
Read how LLMs in Production explains self-supervised learning →
Outlier Detection in Python
Self-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
- How it is used
- Creates training targets from unlabeled data to learn representations for deep learning-based outlier detection.
- How deep it goes
- Explains a rotation-prediction pretext task and its embeddings, then identifies tabular approaches without specifying a complete tabular detection procedure.
- Examples
- Uses four rotations of 100,000 images and considers arithmetic changes and shuffled cells as tabular transformations.
- What you can do after
- Evaluate/understand when automatically generated training targets might yield useful outlier-detection embeddings, especially why tabular data poses difficulties.
Read how Outlier Detection in Python explains self-supervised learning →
The AI Pocket Book
A training approach where a model learns from automatically generated targets derived from the input data itself.
- How it is used
- Generates vast numbers of training examples for language models by extracting targets automatically from existing data.
- How deep it goes
- Explains automatic labeling, input corruption, next-token reconstruction, and the boundary between self-supervised and supervised learning.
- Examples
- Uses an Eiffel Tower sentence, next-token prediction, Common Crawl, and Books3 as concrete training examples or data sources.
- What you can do after
- Evaluate how automatically generated reconstruction targets can scale language-model training while recognizing that performance remains imperfect.
Read how The AI Pocket Book explains self-supervised learning →
Inside Deep Learning: Math, Algorithms, Models
A training setup that derives supervisory signals from the data itself rather than from human-provided labels.
- How it is used
- Reduces dependence on expensive manual labeling while training models on abundant unlabeled data.
- How deep it goes
- Explains implicit targets, autoencoding constraints, sequence losses, and sampling, but defers broader topic-modeling methods.
- Examples
- Uses masked images, shuffled image pieces, reconstructed inputs, and character-level Shakespeare generation from a seed.
- What you can do after
- Apply it via PyTorch by constructing input-derived targets, training with standard losses, and sampling autoregressive outputs.
Read how Inside Deep Learning: Math, Algorithms, Models explains self-supervised learning →
Build a Large Language Model (From Scratch)
A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
- How it is used
- Uses input text to create supervision for LLM pretraining before supervised fine-tuning.
- How deep it goes
- Explains on-the-fly next-word labels and GPT’s autoregressive flow, but defers implementation details.
- Examples
- Uses raw text, sentence prefixes, upcoming words, and GPT text completion as concrete training examples.
- What you can do after
- Apply self-supervision by training a GPT-style model to predict upcoming words from large unlabeled text datasets.
Read how Build a Large Language Model (From Scratch) explains self-supervised learning →
Deep Learning with R, Third Edition
Self-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
- How it is used
- Trains foundation models on massive unlabeled datasets before adapting their learned representations to downstream applications.
- How deep it goes
- Defines input-derived targets and reconstruction objectives, but stops short of detailing optimization or implementation procedures.
- Examples
- Uses next-word prediction, noisy-image reconstruction, and partial-input reconstruction as concrete training examples.
- What you can do after
- Evaluate/understand how unlabeled data can train reusable representations for varied downstream tasks.
Read how Deep Learning with R, Third Edition explains self-supervised learning →
AI-Powered Search
Self-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.
- How it is used
- Pretrains language models and embeddings from large unlabeled text collections before downstream search or language tasks.
- How deep it goes
- Explains token masking, contextual prediction, loss optimization, and embedding formation without detailing complete Transformer internals.
- Examples
- Uses randomly masked sentence tokens and the Transformer encoder as concrete training examples.
- What you can do after
- Evaluate/understand how unlabeled text becomes contextual embeddings through masked-token prediction.
Read how AI-Powered Search explains self-supervised learning →
Generative AI in Action
A machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.
- How it is used
- Supports foundational models that generalize across tasks and power generative AI applications.
- How deep it goes
- Defines the technique and its role in representation learning but does not detail training objectives or implementation steps.
- Examples
- Uses translation, question answering, object recognition, scene understanding, and robotic navigation as application examples.
- What you can do after
- Evaluate/understand how models learn reusable representations from unlabeled multimodal data.
Read how Generative AI in Action explains self-supervised learning →
Related concepts
- language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
- pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
- unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
- foundational modelused by · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
- trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.