Pretraining

As explained in Introduction to Generative AI, Second Edition

The initial training phase in which a model learns general representations or predictive behavior from a large corpus before task-specific adaptation.

Pretraining is the initial phase of building a large language model (LLM). The model learns foundational language patterns and information from an enormous corpus before receiving task-specific refinement. section 2.1 section 2.16

For generative models, pretraining typically uses autoregressive, or causal, token prediction: given previous tokens, the model learns to predict the next word or character. This self-supervised process does not require task-specific labels. section 2.1

Pretraining establishes broad capabilities while also carrying limitations from its data:

• It builds internal representations that support question understanding, contextual interpretation, and closed-book answers from learned information. section 2.2
• It usually draws on open-web material, including public pages from sources such as Wikipedia, Reddit, blogs, news aggregators, and forums. section 2.2
• Its coverage is uneven: English and topics common in the corpus receive stronger representation, while obscure information can produce incorrect or fabricated answers. section 2.2
• Pretraining is followed by post-training, which can target dialogue, instruction following, code generation, or mathematical problem solving. section 2.1 section 2.16

A learned assumption

When asked, “Who was the first president?” an LLM may answer George Washington, implicitly interpreting the question as referring to the United States. That response reflects patterns and geographic imbalances present in its pretraining data, not guaranteed universal understanding. section 2.2

Figure 2.1 The high-level training process for LLMs

Use the high-level training-process figure to distinguish pretraining from the later post-training stages that specialize model behavior. section 2.1

How Introduction to Generative AI, Second Edition covers it

How deep it goes

Explains causal token prediction, corpus learning, and the transition to post-training without detailing optimization internals.

Examples

Uses open-web corpora, Wikipedia, Reddit, English-language imbalance, and the first-president question to illustrate learning and limitations.

What it assumes

Assumes familiarity with transformers, data sources, and token prediction while treating pretraining’s role in the pipeline directly.

What you can do after

Understand how foundational model knowledge arises, why it can be uneven, and how later training specializes behavior.

How it connects

Connects pretraining to token prediction, autoregression, self-supervised learning, post-training, emergent properties, and scaling.

liveBookmanning.com

Read Introduction to Generative AI, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionUnsupervised pretraining is training a model on input data without manually supplied labels before adapting it to downstream tasks.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe initial large-scale training phase used to learn broad language representations before task-specific fine-tuning.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)The initial training phase in which a language model learns general patterns from a large text corpus before task-specific adaptation.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsPretraining is initial training of a model on a large dataset to learn general patterns and relationships before applying it to specific tasks.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA strategy of training a model on a source dataset before adapting it to a downstream task.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)The initial training stage in which a language model learns general statistical structure from large amounts of data before task-specific adaptation.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentPretraining is the initial training of a model on broad data before adapting it to specific tasks or domains.
• 

  liveBookmanning.com
  How Large Language Models WorkPretraining is the initial training of a model on broad data before adapting it to particular tasks or applications.
• 

  liveBookmanning.com
  LLMs in ProductionPretraining is the initial training of a model on a large, broad dataset before adapting it to a specific task or domain.
• 

  liveBookmanning.com
  Sutskever's ListPretraining is the initial training of a model on a broad dataset before task-specific adaptation or evaluation.
• 

  liveBookmanning.com
  Data Without LabelsPretraining is an initial model-training stage that learns general patterns or representations before adaptation to a specific task.
• 

  liveBookmanning.com
  Build a Website with ChatGPTPretraining is the initial training of a model on large amounts of data to learn general patterns before task-specific use.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackPretraining is the initial large-scale optimization of a model on broad data, commonly through next-token prediction.

Compare how 14 books explain pretraining →

Related concepts

• foundational modelproduces · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
• pretrained modelproduces · A pretrained model is a machine-learning model whose parameters were learned through prior training before it is used or adapted for a subsequent task.
• next-token predictionuses · Next-token prediction is the task or training objective of estimating the token that follows a sequence, given its preceding tokens and any conditioning information.
• datasetuses · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.
• zero-shot learningused by · Zero-shot learning is a setting in which a model performs a task or predicts categories without task-specific examples during training, adaptation, or inference, relying on learned knowledge and available task information.
• similarity searchused by · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.