liveBook · Manning

Pretraining

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Pre-training is the initial large-scale training of a model on broad data to learn general patterns before task-specific adaptation or alignment.

Pretraining is the resource-intensive initial phase in which a model learns general patterns, language structures, and world knowledge from massive unlabeled data. It supplies knowledge but not reliable instruction-following, safety refusals, or product-specific steerability. section 5.1.1 section 7.2.1

A typical pretraining workflow:

  • Assemble, clean, and deduplicate an internet-scale corpus containing web pages, books, code, and podcast or video transcripts. section 7.2.1
  • Tokenize the corpus into numerical representations. section 7.2.1
  • Train the model to predict the next-token probability distribution by maximizing log-likelihood and minimizing perplexity. section 7.2.1
  • Compress statistical patterns from the corpus into the model's parameters. section 5.2.1

Pretraining also shapes factual reliability. Frequently represented facts tend to produce stable answers, while rare or absent facts encourage variable pattern completion; consistency is evidence of reliability, not proof. section 5.2.1

Repeated factual questions

Repeatedly asking for the author of “GLU Variants Improve Transformer” can produce the stable answer “Noam Shazeer.” Asking about a less-established paper association may yield different plausible names across attempts, revealing weaker parametric knowledge. section 5.2.1

Figure 7.1 Training of an AI model is typically split into different stages - pretraining and post-training. Mid-training might be split into its own stage operationally.

Use the training-stage diagram to relate pretraining to post-training and the optional operational mid-training stage. section 7.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains corpus preparation, tokenization, next-token likelihood, compressed representations, and the deferred role of mid-training.
Examples
Uses web, books, code, transcripts, and repeated questions about research-paper authorship as concrete training and diagnostic examples.
What you can do after
Understand why pretraining supplies knowledge but not instruction-following, safety refusals, or product-specific steerability.
How it connects
Connects pretraining to supervised fine-tuning, reinforcement learning, parametric memory, and hallucination detection.

Also explained in

Compare how 18 books explain pretraining →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.