Self-supervised learning

As explained in Build a Large Language Model (From Scratch)

A training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.

Self-supervised learning trains a model without human-provided labels by deriving targets from the input data itself. In LLM pretraining, the next word becomes the label for the preceding sequence, allowing training on massive unlabeled text collections. section 1.3 section 1.6

The GPT pretraining workflow connects self-supervision to the model pipeline:

• Prepare large amounts of raw text, with filtering such as removing formatting characters or documents in unknown languages. section 1.3
• Use each sequence to create training pairs: preceding words provide the input, and the upcoming word provides the target. section 1.6
• Train a decoder-style GPT model autoregressively, predicting one word at a time from the words that precede it. section 1.6
• Continue with supervised fine-tuning when task-specific labeled data, instruction-answer pairs, or class labels are available. section 1.3

Next-word prediction

Given a sentence prefix, the model predicts the next word. Repeating this task across large text datasets teaches patterns in how words and phrases fit together, producing a pretrained model that can later support text completion, spelling correction, classification, and translation. section 1.6 section 1.3

Figure 1.7 In the next-word prediction pretraining task for GPT models, the system learns to predict the upcoming word in a sentence by looking at the words that have come before it. This approach helps the model understand how words and phrases typically fit together in language, forming a foundation that can be applied to various other tasks.

Shows how GPT predicts each upcoming word from the preceding sequence and uses the resulting language patterns as a foundation for other tasks. section 1.6

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains on-the-fly next-word labels and GPT’s autoregressive flow, but defers implementation details.

Examples

Uses raw text, sentence prefixes, upcoming words, and GPT text completion as concrete training examples.

What it assumes

Contrasts self-supervised pretraining with conventional supervised learning and introduces labeled fine-tuning afterward.

What you can do after

Apply self-supervision by training a GPT-style model to predict upcoming words from large unlabeled text datasets.

How it connects

Connects pretraining with next-word prediction, GPT, transformer decoders, autoregressive models, and fine-tuning.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionSelf-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
• 

  liveBookmanning.com
  Outlier Detection in PythonSelf-supervised learning is a machine-learning technique that derives training targets from the data itself rather than requiring manually assigned labels.
• 

  liveBookmanning.com
  The AI Pocket BookA training approach where a model learns from automatically generated targets derived from the input data itself.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA training setup that derives supervisory signals from the data itself rather than from human-provided labels.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionSelf-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
• 

  liveBookmanning.com
  AI-Powered SearchSelf-supervised learning is model training in which supervisory targets are derived from the input data itself rather than supplied as external labels.
• 

  liveBookmanning.com
  Generative AI in ActionA machine-learning technique that creates supervisory signals from the data itself to train a model without manually labeled examples.

Compare how 8 books explain self-supervised learning →

Related concepts

• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
• pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
• unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
• foundational modelused by · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
• trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.