Chapter 5: Pretraining on unlabeled data
The chapter introduction connects evaluating generated text to LLM training and explains what trainable weight parameters are and how PyTorch exposes them.
After this chapter you will be able to
- Trainable Weight Parameters. Describe weights as parameters adjusted by learning, and distinguish accessing one PyTorch linear layer's weights with `.weight` from accessing a model's trainable parameters, including biases, with `model.parameters()`.
- GPT text generation cycle. Trace how input text becomes token IDs, passes through GPT as logits, and becomes readable output through tokenization and decoding.
- Need for numerical evaluation. Explain why the untrained model’s generated text motivates a numerical way to evaluate its output.
- Shifted next-token targets. Explain how input sequences are paired with target token IDs and how the model produces a probability distribution for each input position.
- Correct-token probability objective. Explain why evaluation uses probabilities at the target token IDs and what training aims to do to those probabilities.
- Negative average log loss. Trace the manual calculation from target-token probabilities to a single cross entropy loss and state which direction indicates improvement.
- PyTorch cross entropy inputs. Describe the shapes and meanings of the logits and target tensors and how they are supplied to PyTorch’s cross_entropy function.
- Training and validation batches. Describe how the story is divided and loaded into batches for training and validation loss calculations, including the relationship between inputs and targets.
- Loss across data loaders. Trace how the section obtains a loss for each data loader from losses computed on individual batches.
- Initial loss interpretation. Explain why the reported losses are high and what a loss approaching zero would mean in this example.
- Pretraining loop sequence. Trace the basic training workflow: iterate over epochs and batches, reset gradients, calculate loss and new gradients, update weights, and monitor progress.
- Loss and sample monitoring. Distinguish how evaluation losses and generated text samples help track the model during training, including how the section says losses are calculated in evaluation mode.
- Loss divergence and memorization. Explain what the training and validation results show about improvement and overfitting, and why memorization is expected in this run.
- Deterministic token selection. Explain how generate_text_simple selects each new token and why repeated runs with the same starting context produce the same text.
- Greedy versus probabilistic selection. Distinguish greedy decoding from probabilistic next-token sampling. Explain why sampling can select a token other than the most likely one.
- Temperature effects on token probabilities. Explain how dividing logits by a positive temperature before softmax changes token probabilities and the variety of sampled text.
- Top-k sampling mechanism. Trace how top-k sampling selects the tokens with the largest logits, masks the others with negative infinity, and converts the result to probabilities for sampling.
- Combined sampling in generation. Explain that the new generate function combines temperature sampling and top-k sampling, and describe how its example output compares with the earlier generate_simple output.
- Model weight persistence. Explain why saving pretrained weights matters and describe how PyTorch saves and loads them for a new GPTModel instance.
- Optimizer state checkpoints. Explain why continuing pretraining calls for saving and restoring optimizer state as well as model weights.
- Pretrained weights and downloaded data. Explain why using OpenAI’s shared GPT-2 weights avoids retraining on a large corpus, and distinguish the downloaded architecture settings from the weight tensors.
- Matching GPT-2 model configuration. Describe why the GPTModel configuration must match the selected pretrained GPT-2 size, including its 1,024-token context length and query, key, and value bias vectors.
- Weight transfer between implementations. Trace how the loaded params replace a GPTModel instance’s random weights, including shape checks and matching differently named parameters across implementations.
What it builds on
- Implementing multi-head attention with weight splits (chapter 3) · The multi-head attention implementation uses linear layers.
- Coding the GPT model (chapter 4) · The GPTModel provides the model whose parameters will be trained.
- Generating text (chapter 4) · The previously implemented text generation process.
- Converting tokens into token IDs (chapter 2) · Representing text as token IDs.
- Data sampling with a sliding window (chapter 2) · The data loader’s shifted input and target sequences.
- Applying a causal attention mask (chapter 3) · Masking logits with negative infinity before softmax excludes positions by giving them zero probability.
Section by section
Introduction
The chapter introduction connects evaluating generated text to LLM training and explains what trainable weight parameters are and how PyTorch exposes them.
- Evaluation During LLM Training: Explain why evaluating generated text matters during LLM training: measuring its quality is necessary for optimizing the model.
Why it matters: This explains why the chapter evaluates generated text before developing the training process. - Trainable Weight Parameters: Describe weights as parameters adjusted by learning, and distinguish accessing one PyTorch linear layer's weights with `.weight` from accessing a model's trainable parameters, including biases, with `model.parameters()`.
Why it matters: Knowing which values learning adjusts makes the later training code easier to follow.
Key terms: model evaluation, quality of the generated text, optimizing the LLM, weights, weight parameters, parameters, .weight, model.parameters(), biases
5.1 Evaluating generative text models
This section previews a recap of text generation, model evaluation, and training and validation loss calculations.
5.1.1 Using GPT to generate text
Sets up a GPT model for text generation, traces how text becomes tokens and back, and explains why its untrained output needs numerical evaluation.
- GPT text generation cycle: Trace how input text becomes token IDs, passes through GPT as logits, and becomes readable output through tokenization and decoding.
Why it matters: This sequence connects the model’s numerical output to the text a reader actually sees. - Reduced training context length: Explain why this setup reduces the GPT configuration’s context length to 256 tokens rather than using the original 1,024-token setting.
Why it matters: The smaller context explains a practical choice in the model setup used for training. - Need for numerical evaluation: Explain why the untrained model’s generated text motivates a numerical way to evaluate its output.
Why it matters: A measurable evaluation gives training a way to track whether the generated text is improving.
Key terms: tokenizer, token IDs, logits, generate_text_simple, text_to_token_ids, token_ids_to_text, GPT_CONFIG_124M, context length, 256 tokens, 1,024 tokens
5.1.2 Calculating the text generation loss
This section derives next-token generation loss from target-token probabilities, shows how PyTorch computes it, and relates it to perplexity.
- Shifted next-token targets: Explain how input sequences are paired with target token IDs and how the model produces a probability distribution for each input position.
Why it matters: The shifted targets identify which predicted probability to evaluate at every token position. - Correct-token probability objective: Explain why evaluation uses probabilities at the target token IDs and what training aims to do to those probabilities.
Why it matters: A numerical loss can assess how strongly the model favors the right token, not just which token it selected. - Negative average log loss: Trace the manual calculation from target-token probabilities to a single cross entropy loss and state which direction indicates improvement.
Why it matters: These steps explain what the reported loss measures and why a lower value is desirable. - PyTorch cross entropy inputs: Describe the shapes and meanings of the logits and target tensors and how they are supplied to PyTorch’s cross_entropy function.
Why it matters: Passing appropriately shaped raw logits and target IDs lets the built-in function reproduce the manual loss calculation. - Perplexity as uncertainty: Describe how perplexity is obtained from loss and how to interpret a lower value or the example’s large value.
Why it matters: Perplexity gives the loss a more intuitive interpretation in terms of next-token uncertainty.
Key terms: targets, token IDs, logits, softmax, probas, argmax, target token IDs, softmax probability, likelihood, log probabilities
5.1.3 Calculating the training and validation set losses
The section prepares training and validation batches, computes cross-entropy loss over each set, and explains what the initial losses indicate.
- Small dataset rationale: Justify the use of the short story “The Verdict” for this example rather than a large pretraining corpus.
Why it matters: The small corpus makes the loss-calculation example practical without suggesting it is enough to train a capable LLM. - Training and validation batches: Describe how the story is divided and loaded into batches for training and validation loss calculations, including the relationship between inputs and targets.
Why it matters: The loss functions need separate loaders that supply correctly aligned input and next-token target batches. - Loss across data loaders: Trace how the section obtains a loss for each data loader from losses computed on individual batches.
Why it matters: Averaging batch losses supplies a single value for assessing either the training or validation set. - Initial loss interpretation: Explain why the reported losses are high and what a loss approaching zero would mean in this example.
Why it matters: The loss values provide a baseline for judging whether subsequent training improves next-token generation.
Key terms: The Verdict, public domain, 5,145 tokens, training set, validation set, data loaders, max_length, context length, input data (x), cross entropy loss
5.2 Training an LLM
The section presents a simple GPTModel pretraining loop, shows how to monitor training, and interprets the resulting losses and generated text.
- Pretraining loop sequence: Trace the basic training workflow: iterate over epochs and batches, reset gradients, calculate loss and new gradients, update weights, and monitor progress.
Why it matters: This sequence connects the model’s weight updates to the evidence used to track its progress. - Loss and sample monitoring: Distinguish how evaluation losses and generated text samples help track the model during training, including how the section says losses are calculated in evaluation mode.
Why it matters: Loss values and sample text reveal different aspects of whether training is improving the model. - AdamW regularization rationale: Explain why this training loop uses AdamW rather than ordinary Adam.
Why it matters: The optimizer choice is tied to helping the trained model generalize rather than merely fit its training data. - Loss divergence and memorization: Explain what the training and validation results show about improvement and overfitting, and why memorization is expected in this run.
Why it matters: More fluent output and lower training loss do not, by themselves, show that the model generalizes.
Key terms: train_model_simple, epoch, batch, gradients, loss, evaluate_model, generate_and_print_sample, start_context, evaluation mode, training loss
5.3 Decoding strategies to control randomness
The section revisits simple GPT text generation and explains why selecting the highest-probability token produces the same output from the same starting context.
- Deterministic token selection: Explain how generate_text_simple selects each new token and why repeated runs with the same starting context produce the same text.
Why it matters: This explains why the simple generation function does not produce varied text from the same prompt. - Model setup for inference: Describe why the example moves the relatively small model to the CPU and puts it in evaluation mode after training.
Why it matters: These setup choices explain the conditions under which the example generates text.
Key terms: generate_text_simple, token, probability score, start context, inference, evaluation mode, dropout, CPU, GPU
5.3.1 Temperature scaling
This section explains how sampling next tokens from probabilities adds variety to generated text and how temperature scaling changes that variety.
- Greedy versus probabilistic selection: Distinguish greedy decoding from probabilistic next-token sampling. Explain why sampling can select a token other than the most likely one.
Why it matters: This distinction explains how generation can vary even when the model assigns one token the highest probability. - Temperature effects on token probabilities: Explain how dividing logits by a positive temperature before softmax changes token probabilities and the variety of sampled text.
Why it matters: Temperature lets you adjust the trade-off between repeatedly choosing likely tokens and generating more varied, potentially nonsensical text.
Key terms: greedy decoding, torch.argmax, multinomial, probability scores, temperature scaling, logits, softmax, sharper distributions, more uniform distribution
5.3.2 Top-k sampling
This subsection explains how top-k sampling limits next-token choices to the most likely tokens while retaining probabilistic sampling.
- Top-k sampling mechanism: Trace how top-k sampling selects the tokens with the largest logits, masks the others with negative infinity, and converts the result to probabilities for sampling.
Why it matters: This explains how generation can remain random without considering every token in the vocabulary. - Reason for restricting token choices: Explain why top-k sampling is paired with temperature-scaled probabilistic sampling in this section.
Why it matters: The restriction addresses a drawback of seeking more varied output through temperature-scaled sampling alone.
Key terms: top-k sampling, logits, negative infinity, softmax, multinomial function, temperature scaling, probabilistic sampling
5.3.3 Modifying the text generation function
The section combines temperature and top-k sampling in a new text-generation function and compares its output with the earlier simple function.
- Combined sampling in generation: Explain that the new generate function combines temperature sampling and top-k sampling, and describe how its example output compares with the earlier generate_simple output.
Why it matters: This comparison shows the effect of using the new generation function without requiring you to explain implementation details absent from this section.
Key terms: temperature sampling, top-k sampling, generate function, generate_simple function
5.4 Loading and saving model weights in PyTorch
This section explains how to save and reload GPT model weights, prepare a loaded model for inference, and preserve optimizer state when continuing pretraining.
- Model weight persistence: Explain why saving pretrained weights matters and describe how PyTorch saves and loads them for a new GPTModel instance.
Why it matters: Saving weights lets you use a trained model in another session without paying the cost of pretraining again. - Inference evaluation mode: Explain why model.eval() is used for inference after loading a model.
Why it matters: Evaluation mode prevents training-time dropout from affecting generated output. - Optimizer state checkpoints: Explain why continuing pretraining calls for saving and restoring optimizer state as well as model weights.
Why it matters: Restoring optimizer state helps resumed training pick up without discarding the optimizer’s accumulated history.
Key terms: state_dict, torch.save, GPTModel, model weights, model.eval(), evaluation mode, inference, dropout, optimizer state, AdamW
5.5 Loading pretrained weights from OpenAI
This section shows how to download OpenAI’s pretrained GPT-2 weights, configure a matching GPTModel, transfer the weights, and check the result by generating text.
- Pretrained weights and downloaded data: Explain why using OpenAI’s shared GPT-2 weights avoids retraining on a large corpus, and distinguish the downloaded architecture settings from the weight tensors.
Why it matters: The settings describe the model to build, while the weights supply what it learned during pretraining. - Matching GPT-2 model configuration: Describe why the GPTModel configuration must match the selected pretrained GPT-2 size, including its 1,024-token context length and query, key, and value bias vectors.
Why it matters: A matching configuration gives the downloaded parameters the model structure they were trained for. - Weight transfer between implementations: Trace how the loaded params replace a GPTModel instance’s random weights, including shape checks and matching differently named parameters across implementations.
Why it matters: Correctly transferring weights—not merely downloading them—is what turns a newly initialized GPTModel into the pretrained model. - Generated text as loading check: Explain how generated text is used to check whether the pretrained weights were loaded correctly, and what a shape mismatch can reveal earlier.
Why it matters: These checks help catch both incompatible tensor assignments and mapping mistakes that prevent useful text generation.
Key terms: pretrained weights, settings, params, download_and_load_gpt2, model_configs, GPT_CONFIG_124M, NEW_CONFIG, 1,024-token length, bias vectors, GPTModel
5.6 Summary
Summarizes how LLMs generate tokens, how pretraining is assessed and performed, and why loading pretrained weights can be a practical alternative.
Where this is used later
- Loading a pretrained LLM (chapter 7) · Uses GPT text generation.
- Calculating the classification loss and accuracy (chapter 6) · Introduces calculation of text-generation loss before its adaptation to classification.
- Organizing data into training batches (chapter 7) · Calculating text-generation loss from predicted tokens and targets.
- Fine-tuning the LLM on instruction data (chapter 7) · Calculating training and validation losses.
- Fine-tuning the model on supervised data (chapter 6) · Provides the basic training-loop pattern used when fine-tuning.
- Initializing a model with pretrained weights (chapter 6) · Pretraining an LLM.
- (chapter intro) (chapter 6) · Provides the pretrained weights that can be imported into the model.
Concepts explained in this chapter
- GPTModelThe concrete GPT architecture class assembled in this section from embeddings, transformer blocks, normalization, and an output head.
- cross entropy lossA measure of how far a language model's predicted token distribution is from the target token distribution, used to score predictions during training and evaluation.
- generate_text_simpleA simple autoregressive text-generation routine that repeatedly appends the highest-probability next token to a growing context.
- perplexityAn evaluation metric defined as the exponential of cross entropy loss, indicating how uncertain the model is about the next token.
- temperatureA decoding method that rescales next-token probabilities with a temperature to make sampling more or less random.
- embeddingA numerical representation used to encode text or other objects for neural computation.
- pretrainingThe initial training phase in which a language model learns general patterns from a large text corpus before task-specific adaptation.
- train_model_simpleA simple training-loop function used here to train the GPTModel with evaluation and sample generation during pretraining.
- GPT-2A family of transformer-based decoder language models pretrained by OpenAI.
- training datasetThe subset of data used to fit model parameters during development.
- AdamWAn adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.
- top-k samplingA text-generation sampling method that restricts selection to the k highest-scoring next tokens and masks the rest.
- context windowThe number of preceding tokens the model conditions on when predicting the next token.
- samplingA method for choosing next tokens stochastically from their probability distribution instead of always taking the highest-probability choice.
- parameterAn adjustable value in a neural network that is learned during training.
- greedy algorithmA decoding strategy that always selects the highest-probability next token at each generation step.
- logitThe model’s unnormalized output scores over the vocabulary before conversion to probabilities or token selections.
- tokenizerA text-processing component that converts text to token IDs and back, including handling of special tokens and unknown words in this section.
← Chapter 4: Implementing a GPT model from scratch to generate text
Chapter 6: Fine-tuning for classification →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.