liveBook · Manning

Pretraining

As explained in Understanding AI

Pretraining is the initial training of a model on broad data to learn general-purpose representations or statistical patterns before task-specific adaptation.

Why can a language model produce a convincing continuation without necessarily answering your question? Before it becomes an assistant, it is trained to notice what text is statistically likely to come next. This initial phase is pretraining: exposure to a vast collection of text so the model can learn broad patterns of language. section 2.2 section 2.2.1

Training material can include books, articles, websites, Wikipedia pages, discussions, documentation, and source code. These materials are converted into tokens. Given a sequence of tokens, the model predicts the next one; its prediction is compared with the actual next token, and the resulting error makes small adjustments to internal parameters. Repeating this process billions or trillions of times improves its ability to recognize regularities. section 2.2.1 section 2.2.1

A next-token prediction

Given “The performance was unbelievable — the crowd couldn’t…”, a likely next token is “believe.” Early in training, the model may favor unrelated tokens. With repeated correction, it learns that some words, structures, tones, and styles commonly occur together. It is not given hand-written rules for grammar, sarcasm, storytelling, or programming syntax; those patterns emerge indirectly from prediction. section 2.2.1 section 2.2.1

Figure 2.6 A child and a language model learn language through the same fundamental mechanism: exposure and pattern recognition, without explicit instruction.

The child analogy captures the central intuition: extensive exposure and pattern recognition can produce language-like behavior without explicit instruction in grammar rules. section 2.2.1

Pretraining builds a foundation for later stages, not a fully formed helper. A raw pretrained model treats input as text to continue, so several continuations may fit the same prompt. Fine-tuning teaches it to interpret input as an instruction, alignment shapes behavior using human feedback, and specialization redirects the same foundation toward a domain. Throughout the arc, the core mechanism remains next-token prediction. section 2.2 section 2.2.4 section 2.3

The practical takeaway is modest but useful: fluent output reflects learned statistical patterns, not an inherent intention to help or guaranteed correctness. Knowing that distinction helps explain why a model can sound coherent, offer several plausible continuations, or need additional training to behave like a conversational assistant. section 2.2 section 2.3

How Understanding AI covers it

How deep it goes
Explains next-token prediction, error-based parameter updates, and emergent patterns while avoiding mathematical detail.
Examples
Uses a concert-performance completion and a child-language-exposure analogy.
What it assumes
Introduces the mechanism from a plain-language starting point without requiring prior AI terminology.
What you can do after
Evaluate/understand why pretrained models generate plausible continuations rather than inherently follow instructions.
How it connects
Connects pretraining to tokens, fine-tuning, alignment, specialization, and foundation models.

Also explained in

Compare how 18 books explain pretraining →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.