Pretraining
Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
Deep Learning with R, Third Edition
Unsupervised pretraining is training a model on input data without manually supplied labels before adapting it to downstream tasks.
- How it is used
- Uses unlabeled text to prepare embeddings and language models for downstream tasks with limited labeled data.
- How deep it goes
- Walks through CBOW data preparation, model construction, and training, and discusses large-model pretraining tradeoffs.
- Examples
- Uses IMDb reviews for CBOW embedding training and GPT for next-token pretraining and downstream classification.
- What you can do after
- Build a CBOW pretraining model with R and Keras and prepare its embedding for downstream use.
Read how Deep Learning with R, Third Edition explains pretraining →
Deep Learning with Python, Third Edition
The initial large-scale training phase used to learn broad language representations before task-specific fine-tuning.
- How it is used
- Uses unlabeled text to prepare data-hungry sequence models for downstream tasks when labeled examples are limited.
- How deep it goes
- Works through CBOW preprocessing and model construction, and provides a tokenization pipeline for mini-GPT pretraining.
- Examples
- Uses IMDb review embeddings and a mini-GPT trained on a subset of C4.
- What you can do after
- Build word-embedding pretraining from scratch with CBOW and prepare token sequences for Transformer pretraining.
Read how Deep Learning with Python, Third Edition explains pretraining →
Build a Large Language Model (From Scratch)
The initial training phase in which a language model learns general patterns from a large text corpus before task-specific adaptation.
- How it is used
- Positions pretraining as the foundation-building stage before task-specific fine-tuning.
- How deep it goes
- Descends from definition to a PyTorch training loop, loss evaluation, sampling, and weight management, while deferring advanced optimization techniques.
- Examples
- Uses GPT-3’s corpus, a small educational dataset, PyTorch training code, loss plots, and generated continuations as concrete examples.
- What you can do after
- Builds and evaluates a small GPT-like model, or loads pretrained weights for downstream fine-tuning.
Read how Build a Large Language Model (From Scratch) explains pretraining →
Understanding AI
Pretraining is the initial training of a model on broad data to learn general-purpose representations or statistical patterns before task-specific adaptation.
- How it is used
- Frames why a raw language model continues text and why later training is needed for assistant behavior.
- How deep it goes
- Explains next-token prediction, error-based parameter updates, and emergent patterns while avoiding mathematical detail.
- Examples
- Uses a concert-performance completion and a child-language-exposure analogy.
- What you can do after
- Evaluate/understand why pretrained models generate plausible continuations rather than inherently follow instructions.
Time Series Forecasting Using Foundation Models
Pretraining is initial training of a model on a large dataset to learn general patterns and relationships before applying it to specific tasks.
- How it is used
- Uses broad historical data to provide general forecasting capability when a specific task has limited observations.
- How deep it goes
- Explains dataset coverage, windows, horizons, gradient steps, saving, and transfer learning, while stopping short of detailed optimization internals.
- Examples
- Uses monthly M3 data, N-BEATS, Lag-Llama, Moirai, and forecasting for a new product with scarce history.
- What you can do after
- Apply a pretrained model to an unseen forecasting task and save or fine-tune it when coverage or data adequacy is insufficient.
Read how Time Series Forecasting Using Foundation Models explains pretraining →
Inside Deep Learning: Math, Algorithms, Models
A strategy of training a model on a source dataset before adapting it to a downstream task.
- How it is used
- Initializes a model from source-dataset training before adapting it to a downstream task.
- How deep it goes
- Explains pretrained convolutional features and classifier replacement, then summarizes warm and frozen weights without deriving their training mechanics.
- Examples
- Demonstrates ImageNet-pretrained ResNet-18 adapted to cat-versus-dog classification with a replacement final layer.
- What you can do after
- Apply it via torchvision.models by loading pretrained ResNet-18 weights and replacing its final classification layer.
Read how Inside Deep Learning: Math, Algorithms, Models explains pretraining →
Machines that Think
Pretraining is the initial fitting of a model on broad data to acquire general representations or capabilities before task-specific adaptation.
- How it is used
- Establishes a reusable language-model foundation before task-specific adaptation and behavioral shaping.
- How deep it goes
- Explains predictive training and contrasts full and partial adaptation, including catastrophic forgetting.
- Examples
- Uses legal and medical domain adaptation and curated instruction–response pairs to illustrate specialization.
- What you can do after
- Evaluate how a pretrained model supports reuse while recognizing adaptation and grounding limitations.
Retrieval Augmented Generation, The Foundational Ideas
Pretraining is the initial training of a model on broad data before adaptation to a specialized task.
- How it is used
- Connects retrieval to language-model pretraining so the model learns when external evidence improves masked-token prediction.
- How deep it goes
- Descends from masked-token objectives to joint retriever-reader learning and masking choices, but does not detail optimization equations.
- Examples
- Uses a masked monarch-migration sentence, retrieved Wikipedia evidence, and comparisons among SBERT, DPR, and REALM.
- What you can do after
- Evaluate masking choices and distinguish retrieval pretraining based on self-supervised language modeling or labeled question-answer pairs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains pretraining →
Build a Reasoning Model (From Scratch)
The initial training stage in which a language model learns general statistical structure from large amounts of data before task-specific adaptation.
- How it is used
- Builds the general-purpose language model that later supports instruction tuning, preference tuning, or reasoning training.
- How deep it goes
- Explains the data, tokenization, next-token objective, scale, and pipeline placement, but does not detail optimization internals.
- Examples
- Uses tokenized sentences, the Germany–Berlin association, and a staged training-pipeline diagram.
- What you can do after
- Evaluate/understand how next-token prediction builds language capabilities and why later training changes instruction following and reasoning behavior.
Read how Build a Reasoning Model (From Scratch) explains pretraining →
The Art of AI Product Development
Pretraining is the initial training of a model on broad data before adapting it to specific tasks or domains.
- How it is used
- Frames model capabilities and limitations before decisions about customization, user experience, costs, and risks.
- How deep it goes
- Explains contextual next-word prediction and its limits, then stops short of implementation details for training the model.
- Examples
- Uses Mistral and ChatGPT responses to the same conversational prompt, alongside a diagram of contextual word prediction.
- What you can do after
- Evaluate model behavior and discuss whether customization is needed for conversation, instruction following, or specialized product tasks.
Read how The Art of AI Product Development explains pretraining →
How Large Language Models Work
Pretraining is the initial training of a model on broad data before adapting it to particular tasks or applications.
- How it is used
- Establishes broad model capabilities before fine-tuning constrains behavior for a particular task.
- How deep it goes
- Explains the training pipeline and rationale for broad data, but stops short of coding an LLM from scratch.
- Examples
- Uses web pages, general documents, and a car-salesperson scenario involving sports equipment and accessibility.
- What you can do after
- Understand how broad pretraining and narrower fine-tuning divide general knowledge from task-specific behavior.
Read how How Large Language Models Work explains pretraining →
Introduction to Generative AI, Second Edition
The initial training phase in which a model learns general representations or predictive behavior from a large corpus before task-specific adaptation.
- How it is used
- Frames pretraining as the foundational stage of a multistage pipeline for developing capable language models.
- How deep it goes
- Explains causal token prediction, corpus learning, and the transition to post-training without detailing optimization internals.
- Examples
- Uses open-web corpora, Wikipedia, Reddit, English-language imbalance, and the first-president question to illustrate learning and limitations.
- What you can do after
- Understand how foundational model knowledge arises, why it can be uneven, and how later training specializes behavior.
Read how Introduction to Generative AI, Second Edition explains pretraining →
LLMs in Production
Pretraining is the initial training of a model on a large, broad dataset before adapting it to a specific task or domain.
- How it is used
- Establishes broad language capabilities before adapting an LLM to a specific task or domain.
- How deep it goes
- Explains corpus purpose, learned capabilities, cost, and transfer workflow, but stops short of detailed research internals.
- Examples
- Uses Wikipedia, books, the internet, and a from-scratch Llama 3 project as training contexts.
- What you can do after
- Build it from scratch as part of an end-to-end LLM project, then prepare, adapt, and deploy the model.
LLM Evaluation and Alignment, The Foundational Ideas
Pre-training is the initial large-scale training of a model on broad data to learn general patterns before task-specific adaptation or alignment.
- How it is used
- Provides foundational knowledge before supervised fine-tuning and reinforcement learning address steerability and preferences.
- How deep it goes
- Explains corpus preparation, tokenization, next-token likelihood, compressed representations, and the deferred role of mid-training.
- Examples
- Uses web, books, code, transcripts, and repeated questions about research-paper authorship as concrete training and diagnostic examples.
- What you can do after
- Understand why pretraining supplies knowledge but not instruction-following, safety refusals, or product-specific steerability.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains pretraining →
Sutskever's List
Pretraining is the initial training of a model on a broad dataset before task-specific adaptation or evaluation.
- How it is used
- Uses layer-wise unsupervised training to initialize deep networks before end-to-end supervised fine-tuning.
- How deep it goes
- Explains layer-by-layer training, unrolling, and fine-tuning, while leaving implementation details unspecified.
- Examples
- Illustrates the procedure by feeding each layer’s output into the next before unrolling the network.
- What you can do after
- Understand how layer-wise initialization supported deep-network training and why later work challenged its necessity.
Data Without Labels
Pretraining is an initial model-training stage that learns general patterns or representations before adaptation to a specific task.
- How it is used
- Uses unlabeled data to initialize deep networks before supervised adaptation to prediction tasks.
- How deep it goes
- Explains layer-wise RBM training, contrastive divergence, Gibbs sampling, and subsequent backpropagation-based fine-tuning.
- Examples
- Uses image recognition, speech recognition, dimensionality reduction, generative modeling, classification, and regression as application examples.
- What you can do after
- Understand how to sequence unsupervised layer training and supervised fine-tuning when developing deep belief networks.
Build a Website with ChatGPT
Pretraining is the initial training of a model on large amounts of data to learn general patterns before task-specific use.
- How it is used
- Explains how GPT gains broad language and programming abilities before responding to requests such as creating web pages.
- How deep it goes
- Defines exposure, learned patterns, and next-word prediction, but stops short of describing training algorithms or model internals.
- Examples
- Uses sentence formation, next-word prediction, and exposure to programming code as concrete examples of learned patterns.
- What you can do after
- Evaluate/understand why GPT can generate text and web-page code after learning broad patterns from large datasets.
Read how Build a Website with ChatGPT explains pretraining →
Reinforcement Learning from Human Feedback
Pretraining is the initial large-scale optimization of a model on broad data, commonly through next-token prediction.
- How it is used
- Establishes the broad capability base that later post-training turns into more useful conversational behavior.
- How deep it goes
- Contrasts pretraining with post-training and next-token prediction, but does not detail datasets, optimization procedures, or model internals.
- Examples
- Uses the OLMoE Instruct version comparison to show how post-training can improve a mostly unchanged pretrained foundation.
- What you can do after
- Evaluate pretraining as capability creation and distinguish it from post-training’s elicitation of that capability.
Read how Reinforcement Learning from Human Feedback explains pretraining →
Also covered in
- Inside AI
- Deep Learning with JAX
- Grokking AI Applications
- Hugging Face in Action
- Data Analysis with LLMs
- The Complete Obsolete Guide to Generative AI
- Natural Language Processing in Action, Second Edition
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- GPT modelA GPT model is a generative, autoregressive Transformer language model in the family developed by OpenAI.
Related concepts
- foundation modelproduces · A foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
- next-word predictionuses · Next-token prediction is a language-modeling task or objective in which a model predicts the token that follows a given context.
- pretrained modelproduces · A pretrained model is a machine-learning model whose parameters were learned through prior training before it is used for inference or adapted to a later task.
- datasetuses · A dataset is an organized collection of related data items, records, observations, or examples treated as a unit for storage, processing, or analysis.
- fine-tuningcontrasts with · Fine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- similarity searchused by · Similarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
Questions about pretraining
What is pretraining?
Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
Which Manning books explain pretraining?
Deep Learning with R, Third Edition, Deep Learning with Python, Third Edition, Build a Large Language Model (From Scratch), Understanding AI, Time Series Forecasting Using Foundation Models, Inside Deep Learning: Math, Algorithms, Models, Machines that Think, Retrieval Augmented Generation, The Foundational Ideas and 10 more.
What should I know before learning pretraining?
Manning books that teach pretraining usually introduce language model, fine-tuning, transformer architecture and GPT model first.
What is the difference between pretraining and fine-tuning?
Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains. Fine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
What is pretraining also called?
Pretraining is also called pre-training or pretrain.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

















