Cover of Build a Large Language Model (From Scratch)

What you will learn in Build a Large Language Model (From Scratch)

by Sebastian Raschka

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: Understanding large language models

The introduction contrasts LLMs with earlier NLP models, clarifies what their apparent understanding means, and attributes their broader capabilities to transformer architectures and large-scale text training.

1.1 What is an LLM? · 1.2 Applications of LLMs · 1.3 Stages of building and using LLMs · 1.4 Introducing the transformer architecture · 1.5 Utilizing large datasets · 1.6 A closer look at the GPT architecture · 1.7 Building a large language model · 1.8 Summary

What you learn in chapter 1 →

Chapter 2: Working with text data

This introduction explains why text data must be prepared before pretraining an LLM and previews the chapter’s data pipeline.

2.1 Understanding word embeddings · 2.2 Tokenizing text · 2.3 Converting tokens into token IDs · 2.4 Adding special context tokens · 2.5 Byte pair encoding · 2.6 Data sampling with a sliding window · 2.7 Creating token embeddings · 2.8 Encoding word positions · 2.9 Summary

What you learn in chapter 2 →

Chapter 3: Coding attention mechanisms

The introduction connects prepared text embeddings to attention as an integral part of the LLM architecture.

3.1 The problem with modeling long sequences · 3.2 Capturing data dependencies with attention mechanisms · 3.3 Attending to different parts of the input with self-attention · 3.3.1 A simple self-attention mechanism without trainable weights · 3.3.2 Computing attention weights for all input tokens · 3.4 Implementing self-attention with trainable weights · 3.4.1 Computing the attention weights step by step · 3.4.2 Implementing a compact self-attention Python class · 3.5 Hiding future words with causal attention · 3.5.1 Applying a causal attention mask · 3.5.2 Masking additional attention weights with dropout · 3.5.3 Implementing a compact causal attention class · 3.6 Extending single-head attention to multi-head attention · 3.6.1 Stacking multiple single-head attention layers · 3.6.2 Implementing multi-head attention with weight splits · 3.7 Summary

What you learn in chapter 3 →

Chapter 4: Implementing a GPT model from scratch to generate text

Introduces the plan to assemble previously coded multi-head attention with other building blocks into a GPT-like model.

4.1 Coding an LLM architecture · 4.2 Normalizing activations with layer normalization · 4.3 Implementing a feed forward network with GELU activations · 4.4 Adding shortcut connections · 4.5 Connecting attention and linear layers in a transformer block · 4.6 Coding the GPT model · 4.7 Generating text · 4.8 Summary

What you learn in chapter 4 →

Chapter 5: Pretraining on unlabeled data

The chapter introduction connects evaluating generated text to LLM training and explains what trainable weight parameters are and how PyTorch exposes them.

5.1 Evaluating generative text models · 5.1.1 Using GPT to generate text · 5.1.2 Calculating the text generation loss · 5.1.3 Calculating the training and validation set losses · 5.2 Training an LLM · 5.3 Decoding strategies to control randomness · 5.3.1 Temperature scaling · 5.3.2 Top-k sampling · 5.3.3 Modifying the text generation function · 5.4 Loading and saving model weights in PyTorch · 5.5 Loading pretrained weights from OpenAI · 5.6 Summary

What you learn in chapter 5 →

Chapter 6: Fine-tuning for classification

The chapter introduces classification fine-tuning as adapting a pretrained LLM to a specific task, using spam detection as its example.

6.1 Different categories of fine-tuning · 6.2 Preparing the dataset · 6.3 Creating data loaders · 6.4 Initializing a model with pretrained weights · 6.5 Adding a classification head · 6.6 Calculating the classification loss and accuracy · 6.7 Fine-tuning the model on supervised data · 6.8 Using the LLM as a spam classifier · 6.9 Summary

What you learn in chapter 6 →

Chapter 7: Fine-tuning to follow instructions

The chapter introduction contrasts instruction fine-tuning with the earlier classification task and explains why following instructions matters for conversational LLMs.

7.1 Introduction to instruction fine-tuning · 7.2 Preparing a dataset for supervised instruction fine-tuning · 7.3 Organizing data into training batches · 7.4 Creating data loaders for an instruction dataset · 7.5 Loading a pretrained LLM · 7.6 Fine-tuning the LLM on instruction data · 7.7 Extracting and saving responses · 7.8 Evaluating the fine-tuned LLM · 7.9 Conclusions · 7.9.1 What’s next? · 7.9.2 Staying up to date in a fast-moving field · 7.9.3 Final words · 7.10 Summary

What you learn in chapter 7 →

Concepts explained in this book

fine-tuningFurther training of a pretrained model on a smaller task-specific dataset to improve downstream performance.
embeddingA numerical representation used to encode text or other objects for neural computation.
GPTModelThe concrete GPT architecture class assembled in this section from embeddings, transformer blocks, normalization, and an output head.
attentionA neural-network mechanism that weights input tokens or values by relevance so the model can focus on important context.
batch sizeThe number of training examples processed together in one update or evaluation step.
GPT-2A family of transformer-based decoder language models pretrained by OpenAI.
token idAn integer identifier assigned to a token by a vocabulary for use in model input and output processing.
context vectorA vector representation formed by combining input information with attention weights to summarize relevant context for a token.
cross entropy lossA measure of how far a language model's predicted token distribution is from the target token distribution, used to score predictions during training and evaluation.
layer normalizationA normalization technique that stabilizes neural network training by normalizing activations to have more consistent statistics across layers.
language modelA large neural language model trained on text to understand and generate language, typically using transformer self-attention.
pretrainingThe initial training phase in which a language model learns general patterns from a large text corpus before task-specific adaptation.
GPT modelA family of decoder-only generative pretrained transformer language models.
attention scoreThe numeric values computed to quantify how strongly one token should attend to another in attention mechanisms.
attention weightA normalized relevance coefficient derived from attention scores that determines how much each value contributes to the context vector.
training datasetThe subset of data used to fit model parameters during development.
Alpaca datasetA dataset of instruction-style input-output examples used to adapt a language model to follow natural-language instructions.
generate_text_simpleA simple autoregressive text-generation routine that repeatedly appends the highest-probability next token to a growing context.
byte pair encodingA subword tokenization method that iteratively merges frequent symbol pairs to build a vocabulary for text representation.
next-token predictionA language-model training objective in which the model predicts the next token or word from preceding context.
datasetA collection of examples used for training or experimentation.
positional embeddingA position-encoding method that assigns a distinct embedding to each specific sequence position.
end of text tokenA special token used to mark the end of a text sequence or response in GPT-style tokenization.
multi-head attentionAn attention technique that computes several independent attention heads in parallel and combines their outputs to capture different relationships in the same sequence.
parameterAn adjustable value in a neural network that is learned during training.
token limitThe fixed number of token IDs included in each sampled input sequence for the loaders in this section.
maskA training-time method that prevents the model from attending to certain input positions, such as padding tokens.
tokenizationThe process of splitting raw text into tokens for further processing by a language model.
tokenA discrete unit of text that a language model processes, roughly corresponding to words or punctuation marks in this section.
perplexityAn evaluation metric defined as the exponential of cross entropy loss, indicating how uncertain the model is about the next token.
collate functionA data-loading function that merges a list of individual samples into a single batch, optionally applying padding and other formatting.
spam classificationThe text classification task of deciding whether an email or message is spam or legitimate.
temperatureA decoding method that rescales next-token probabilities with a temperature to make sampling more or less random.
feedforward networkThe transformer subcomponent that applies position-wise nonlinear transformations after attention.
tokenizerA text-processing component that converts text to token IDs and back, including handling of special tokens and unknown words in this section.
train_model_simpleA simple training-loop function used here to train the GPTModel with evaluation and sample generation during pretraining.
low-rank adaptationA parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
OllamaA local application and serving wrapper for running large language models for inference through a command line or API.
qualityA numeric evaluation result used here to summarize how well the fine-tuned model's responses are judged by another model.
DataLoaderA component that produces batches of data for model training or experimentation, typically by sampling and collating examples.
accuracyThe fraction or percentage of predictions that match the correct class labels in a classification task.
logitThe model’s unnormalized output scores over the vocabulary before conversion to probabilities or token selections.
self-supervised learningA training approach in which the model derives supervision signals from the input data itself rather than from human-provided labels.
input-target pairA paired training example consisting of an input sequence and the corresponding target sequence shifted for next-token prediction.
sliding windowA data-sampling method that moves a fixed-size window across text to create overlapping training examples.
vocabularyA mapping between unique text tokens and unique integer identifiers used to encode and decode text.
transformer architecturea neural network architecture based on self-attention mechanisms that enables processing of sequential data and underpins many large language models.
keyThe attention-side vector associated with an input token and compared with a query to determine relevance.
valueThe attention content vector combined according to attention weights to form the context output.
greedy algorithmA decoding strategy that always selects the highest-probability next token at each generation step.
linear output layerThe final linear projection that maps transformer outputs into vocabulary-sized logits for next-token prediction.
weight tyingA parameter-sharing technique that reuses the token embedding weights in the output layer to reduce the effective parameter count.
out-of-vocabulary wordA placeholder token used by some tokenizers for out-of-vocabulary words, which BPE avoids here by subword decomposition.
Order propertyThe location of a token or element within a sequence, used here as the basis for relating elements in self-attention.
last tokenThe final token position in an input sequence, used here as the summary position for classification.
output layerThe final layer of a model that produces prediction scores or class outputs from internal representations.
few-shot learningThe ability to learn a new task from only a few examples rather than extensive training data.
strideThe step size by which the sliding window advances between sampled training examples.
samplingA method for choosing next tokens stochastically from their probability distribution instead of always taking the highest-probability choice.
classifierThe task of classifying text into predefined categories.
CausalAttentionA causal single-head attention class that computes attention while preserving autoregressive token order.
MultiHeadAttentionA single attention module that computes multiple attention heads by reshaping projected queries, keys, and values within one implementation.
MultiHeadAttentionWrapperA wrapper class that implements multi-head attention by combining several separate single-head attention modules.
AdamWAn adaptive optimization algorithm that maintains historical information to adjust learning rates for neural-network parameters.
GELUA smooth activation function defined as x times the Gaussian CDF, used here in transformer feed-forward submodules.
relative positional embeddingA position-encoding method that represents how far apart tokens are rather than their exact indices.
context windowThe number of preceding tokens the model conditions on when predicting the next token.
conversational performanceThe degree to which an LLM communicates in a human-like, context-aware, coherent manner in dialogue.
generateA function used to produce model outputs from input prompts by running generation over a fine-tuned language model.
vanishing gradient problemA training difficulty in deep neural networks where gradients shrink as they propagate backward, hindering learning in early layers.
top-k samplingA text-generation sampling method that restricts selection to the k highest-scoring next tokens and masks the rest.
subword tokenizationA token fragment smaller than a full word, used by subword tokenizers to represent rare or unseen words.
PyTorchThe deep learning library used here to normalize attention scores along a specified tensor dimension with softmax.
BERTA transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.
zero-shot learningA learning setup where a model handles a task without having seen any task-specific examples beforehand.
llama.cppA C/C++ library for efficiently running large language models, used here as the engine behind Ollama.
trainingThe process of optimizing model parameters using data so the model learns the desired behavior.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.