Pretraining is the initial training of a model on broad data before it is adapted to a particular task. In language models, it builds general linguistic capabilities that can be reused across applications rather than learned separately for each one. section 3.5.6 section 4.3.1
A language model can learn from raw text through a general predictive objective: it predicts parts of the text and adjusts its parameters according to the difference between its predictions and the actual text. Across large collections of documents, it acquires patterns of grammar, facts, style, reasoning traces, and association. These patterns provide a foundation, but predicting text is not the same as producing the controlled responses an application requires. section 4.3.1 section 7.1.1
After pretraining, the model can be specialized through further learning. Full fine-tuning adjusts all its parameters; partial adaptation changes only some parameters or adds trainable components alongside a frozen model. Supervised fine-tuning can use curated instruction–response pairs to make desired kinds of responses more likely. These methods build on the pretrained foundation rather than replacing its predictive core. section 4.3.1 section 7.1.1
Figure 4.13 Pretraining and model adaptation. Models are first trained on broad data, then specialized through full or partial parameter updates.

The figure contrasts broad pretraining with specialization through full or partial parameter updates. section 4.3.1
Specializing for a domain
A model pretrained on broad text can be further adapted with a smaller collection of legal or medical text, instead of training a separate model from scratch for each domain. section 3.5.6
Reusing a pretrained model reduces the need for repeated task-specific training and large labeled datasets, although large-scale training still carries substantial upfront economic and energy costs. Adaptation also has a trade-off: shifting too far toward a specialized dataset or objective can weaken earlier capabilities, a problem called catastrophic forgetting. section 3.5.6 section 4.8 section 4.3.1
A model’s fluent output should also be interpreted cautiously. Text indirectly encodes human experience, so a model can appear knowledgeable without directly perceiving the world. Tasks requiring visual, auditory, or temporal grounding can expose the limits of text-only pretraining. section 4.8

















