Pretraining is the resource-intensive initial phase in which a model learns general patterns, language structures, and world knowledge from massive unlabeled data. It supplies knowledge but not reliable instruction-following, safety refusals, or product-specific steerability. section 5.1.1 section 7.2.1
A typical pretraining workflow:
- Assemble, clean, and deduplicate an internet-scale corpus containing web pages, books, code, and podcast or video transcripts. section 7.2.1
- Tokenize the corpus into numerical representations. section 7.2.1
- Train the model to predict the next-token probability distribution by maximizing log-likelihood and minimizing perplexity. section 7.2.1
- Compress statistical patterns from the corpus into the model's parameters. section 5.2.1
Pretraining also shapes factual reliability. Frequently represented facts tend to produce stable answers, while rare or absent facts encourage variable pattern completion; consistency is evidence of reliability, not proof. section 5.2.1
Repeated factual questions
Repeatedly asking for the author of “GLU Variants Improve Transformer” can produce the stable answer “Noam Shazeer.” Asking about a less-established paper association may yield different plausible names across attempts, revealing weaker parametric knowledge. section 5.2.1
Figure 7.1 Training of an AI model is typically split into different stages - pretraining and post-training. Mid-training might be split into its own stage operationally.
Use the training-stage diagram to relate pretraining to post-training and the optional operational mid-training stage. section 7.2.1

















