Pretraining
Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
Deep Learning with R, Third Edition
Unsupervised pretraining is training a model on input data without manually supplied labels before adapting it to downstream tasks.
- How it is used
- Primes data-hungry text models on unlabeled data before downstream classification.
- How deep it goes
- Works through CBOW data preparation and embedding training, then motivates large-scale Transformer pretraining without reproducing a billion-parameter run.
- Examples
- Uses unlabeled IMDb reviews for CBOW and pretrained RoBERTa for review classification.
- What you can do after
- Build it from scratch as a CBOW word embedding and understand how its learned parameters support downstream tasks.
Read how Deep Learning with R, Third Edition explains pretraining →
Deep Learning with Python, Third Edition
The initial large-scale training phase used to learn broad language representations before task-specific fine-tuning.
- How it is used
- Uses unlabeled text to prepare model parameters before transferring them to a labeled text task.
- How deep it goes
- Works through CBOW embedding training and mini-GPT data preparation, while treating full-scale Transformer pretraining as prohibitively expensive.
- Examples
- Uses IMDb review windows for CBOW and tokenized C4 documents for mini-GPT pretraining.
- What you can do after
- Build it from scratch with a CBOW embedding exercise or a mini-GPT preprocessing pipeline; evaluate when a pretrained model is more practical.
Read how Deep Learning with Python, Third Edition explains pretraining →
Build a Large Language Model (From Scratch)
The initial training phase in which a language model learns general patterns from a large text corpus before task-specific adaptation.
- How it is used
- Positions pretraining as the foundation-building stage before task-specific fine-tuning.
- How deep it goes
- Descends from definition to a PyTorch training loop, loss evaluation, sampling, and weight management, while deferring advanced optimization techniques.
- Examples
- Uses GPT-3’s corpus, a small educational dataset, PyTorch training code, loss plots, and generated continuations as concrete examples.
- What you can do after
- Builds and evaluates a small GPT-like model, or loads pretrained weights for downstream fine-tuning.
Read how Build a Large Language Model (From Scratch) explains pretraining →
Time Series Forecasting Using Foundation Models
Pretraining is initial training of a model on a large dataset to learn general patterns and relationships before applying it to specific tasks.
- How it is used
- Uses broad historical data to provide general forecasting capability when a specific task has limited observations.
- How deep it goes
- Explains dataset coverage, windows, horizons, gradient steps, saving, and transfer learning, while stopping short of detailed optimization internals.
- Examples
- Uses monthly M3 data, N-BEATS, Lag-Llama, Moirai, and forecasting for a new product with scarce history.
- What you can do after
- Apply a pretrained model to an unseen forecasting task and save or fine-tune it when coverage or data adequacy is insufficient.
Read how Time Series Forecasting Using Foundation Models explains pretraining →
Inside Deep Learning: Math, Algorithms, Models
A strategy of training a model on a source dataset before adapting it to a downstream task.
- How it is used
- Deploys source-trained weights to accelerate adaptation of a model to a new task, especially when labeled target data is scarce.
- How deep it goes
- Explains model surgery, weight initialization, feature intuition, and warm or frozen training, but stops short of formal guarantees.
- Examples
- Uses ImageNet-trained ResNet-18, cat-versus-dog classification, replaced linear heads, and visualized convolutional filters.
- What you can do after
- Apply it via torchvision.models by loading pretrained weights, replacing the task-specific head, and training with warm or frozen parameters.
Read how Inside Deep Learning: Math, Algorithms, Models explains pretraining →
Build a Reasoning Model (From Scratch)
The initial training stage in which a language model learns general statistical structure from large amounts of data before task-specific adaptation.
- How it is used
- Builds the general-purpose language model that later supports instruction tuning, preference tuning, or reasoning training.
- How deep it goes
- Explains the data, tokenization, next-token objective, scale, and pipeline placement, but does not detail optimization internals.
- Examples
- Uses tokenized sentences, the Germany–Berlin association, and a staged training-pipeline diagram.
- What you can do after
- Evaluate/understand how next-token prediction builds language capabilities and why later training changes instruction following and reasoning behavior.
Read how Build a Reasoning Model (From Scratch) explains pretraining →
The Art of AI Product Development
Pretraining is the initial training of a model on broad data before adapting it to specific tasks or domains.
- How it is used
- Frames model capabilities and limitations before decisions about customization, user experience, costs, and risks.
- How deep it goes
- Explains contextual next-word prediction and its limits, then stops short of implementation details for training the model.
- Examples
- Uses Mistral and ChatGPT responses to the same conversational prompt, alongside a diagram of contextual word prediction.
- What you can do after
- Evaluate model behavior and discuss whether customization is needed for conversation, instruction following, or specialized product tasks.
Read how The Art of AI Product Development explains pretraining →
How Large Language Models Work
Pretraining is the initial training of a model on broad data before adapting it to particular tasks or applications.
- How it is used
- Establishes broad model capabilities before fine-tuning constrains behavior for a particular task.
- How deep it goes
- Explains the training pipeline and rationale for broad data, but stops short of coding an LLM from scratch.
- Examples
- Uses web pages, general documents, and a car-salesperson scenario involving sports equipment and accessibility.
- What you can do after
- Understand how broad pretraining and narrower fine-tuning divide general knowledge from task-specific behavior.
Read how How Large Language Models Work explains pretraining →
Introduction to Generative AI, Second Edition
The initial training phase in which a model learns general representations or predictive behavior from a large corpus before task-specific adaptation.
- How it is used
- Frames pretraining as the foundational stage of a multistage pipeline for developing capable language models.
- How deep it goes
- Explains causal token prediction, corpus learning, and the transition to post-training without detailing optimization internals.
- Examples
- Uses open-web corpora, Wikipedia, Reddit, English-language imbalance, and the first-president question to illustrate learning and limitations.
- What you can do after
- Understand how foundational model knowledge arises, why it can be uneven, and how later training specializes behavior.
Read how Introduction to Generative AI, Second Edition explains pretraining →
LLMs in Production
Pretraining is the initial training of a model on a large, broad dataset before adapting it to a specific task or domain.
- How it is used
- Establishes broad language capabilities before adapting an LLM to a specific task or domain.
- How deep it goes
- Explains corpus purpose, learned capabilities, cost, and transfer workflow, but stops short of detailed research internals.
- Examples
- Uses Wikipedia, books, the internet, and a from-scratch Llama 3 project as training contexts.
- What you can do after
- Build it from scratch as part of an end-to-end LLM project, then prepare, adapt, and deploy the model.
Sutskever's List
Pretraining is the initial training of a model on a broad dataset before task-specific adaptation or evaluation.
- How it is used
- Uses layer-wise unsupervised training to initialize deep networks before end-to-end supervised fine-tuning.
- How deep it goes
- Explains layer-by-layer training, unrolling, and fine-tuning, while leaving implementation details unspecified.
- Examples
- Illustrates the procedure by feeding each layer’s output into the next before unrolling the network.
- What you can do after
- Understand how layer-wise initialization supported deep-network training and why later work challenged its necessity.
Data Without Labels
Pretraining is an initial model-training stage that learns general patterns or representations before adaptation to a specific task.
- How it is used
- Uses unlabeled data to initialize deep networks before supervised adaptation to prediction tasks.
- How deep it goes
- Explains layer-wise RBM training, contrastive divergence, Gibbs sampling, and subsequent backpropagation-based fine-tuning.
- Examples
- Uses image recognition, speech recognition, dimensionality reduction, generative modeling, classification, and regression as application examples.
- What you can do after
- Understand how to sequence unsupervised layer training and supervised fine-tuning when developing deep belief networks.
Build a Website with ChatGPT
Pretraining is the initial training of a model on large amounts of data to learn general patterns before task-specific use.
- How it is used
- Explains how GPT gains broad language and programming abilities before responding to requests such as creating web pages.
- How deep it goes
- Defines exposure, learned patterns, and next-word prediction, but stops short of describing training algorithms or model internals.
- Examples
- Uses sentence formation, next-word prediction, and exposure to programming code as concrete examples of learned patterns.
- What you can do after
- Evaluate/understand why GPT can generate text and web-page code after learning broad patterns from large datasets.
Read how Build a Website with ChatGPT explains pretraining →
Reinforcement Learning from Human Feedback
Pretraining is the initial large-scale optimization of a model on broad data, commonly through next-token prediction.
- How it is used
- Establishes the broad capability base that later post-training turns into more useful conversational behavior.
- How deep it goes
- Contrasts pretraining with post-training and next-token prediction, but does not detail datasets, optimization procedures, or model internals.
- Examples
- Uses the OLMoE Instruct version comparison to show how post-training can improve a mostly unchanged pretrained foundation.
- What you can do after
- Evaluate pretraining as capability creation and distinguish it from post-training’s elicitation of that capability.
Read how Reinforcement Learning from Human Feedback explains pretraining →
Related concepts
- foundational modelproduces · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
- pretrained modelproduces · A pretrained model is a machine-learning model whose parameters were learned through prior training before it is used or adapted for a subsequent task.
- next-token predictionuses · Next-token prediction is the task or training objective of estimating the token that follows a sequence, given its preceding tokens and any conditioning information.
- datasetuses · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.
- zero-shot learningused by · Zero-shot learning is a setting in which a model performs a task or predicts categories without task-specific examples during training, adaptation, or inference, relying on learned knowledge and available task information.
- similarity searchused by · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.