Chapter 2: Working with text data

Introduction

This introduction explains why text data must be prepared before pretraining an LLM and previews the chapter’s data pipeline.

• Pretraining data prerequisite: Explain that LLMs are pretrained on large amounts of text using a next-word prediction task, and that the training dataset must be prepared before an LLM can be implemented and trained.
  Why it matters: It connects the chapter’s focus on text data to the task used to pretrain an LLM.

Key terms: pretraining, next-word prediction task, training dataset

2.1 Understanding word embeddings

This section explains why LLMs need embeddings, what word embeddings represent, and how their training and dimensionality affect their use.

• Purpose of word embeddings: Explain why an LLM cannot use raw text directly and how an embedding makes text usable by a neural network.
  Why it matters: This is the reason text needs a numeric representation before an LLM can process it.
• Context and word similarity: Describe the idea behind Word2Vec and what nearby word vectors in its two-dimensional visualization illustrate.
  Why it matters: Word2Vec gives a concrete example of how relationships between words can appear in an embedding space.
• LLM-trained input embeddings: Distinguish commonly used LLM input embeddings from using pretrained Word2Vec embeddings, and explain the advantage of training them with the LLM.
  Why it matters: It explains why an LLM can learn input representations suited to the data and task it is trained on.
• Embedding dimensionality tradeoff: Explain the tradeoff in choosing an embedding dimension and why the section uses a two-dimensional plot to show embeddings.
  Why it matters: It helps you interpret the simple plot without mistaking it for the size of embeddings typically used in LLMs.

Key terms: raw text, categorical, embedding, continuous-valued vectors, Word2Vec, context, word embeddings, pretrained models, input layer, updated during training

2.2 Tokenizing text

This section builds a simple tokenizer that separates words and punctuation in a short story, then explains choices about capitalization and whitespace.

• Tokens before embeddings: Explain what tokenization produces and why it is part of preparing text for an LLM.
  Why it matters: This establishes what the later text-processing steps operate on.
• Progressive punctuation separation: Trace how the simple regular-expression tokenizer is refined and how its output is checked on the short story.
  Why it matters: The refinements show why a first split is not enough to produce the intended tokens.
• Preserving capitalization in tokens: Justify why the example does not lowercase all the text before tokenization.
  Why it matters: Keeping case preserves information the model may use.
• Whitespace removal tradeoff: Explain why this example removes whitespace tokens and when retaining them could matter.
  Why it matters: Whether to keep spaces depends on what information the application needs to preserve.

Key terms: tokens, tokenization, embeddings, re.split, whitespace, punctuation characters, special characters, capitalization, proper nouns, common nouns

2.3 Converting tokens into token IDs

The section builds a vocabulary to map text tokens to integer IDs, uses a tokenizer to encode and decode text, and shows why a vocabulary built from limited training text can fail on new input.

• Vocabulary and token IDs: Explain how a vocabulary assigns unique integer IDs to the unique tokens in training text, and why token IDs are needed before embedding vectors.
  Why it matters: The vocabulary makes it possible to represent text tokens as numbers the model can use.
• Encoding and decoding text: Trace how the tokenizer's encode method turns text into token IDs using the vocabulary, and how its decode method uses an inverse mapping to turn those IDs back into text.
  Why it matters: Both directions are needed to prepare text as model input and turn numerical output back into readable text.
• Limits of a small vocabulary: Explain why encoding a new sample containing “Hello” fails with the story-based vocabulary and what this demonstrates about training-text coverage.
  Why it matters: A tokenizer that works on its source text may still fail when new text contains unseen tokens.

Key terms: vocabulary, token IDs, embedding vectors, encode, decode, inverse vocabulary, SimpleTokenizerV1, unknown words, training set

2.4 Adding special context tokens

This section adds tokens for unknown words and boundaries between unrelated texts to a simple tokenizer, then contrasts those choices with other special tokens and GPT tokenization.

• Unknown word handling: Explain how SimpleTokenizerV2 handles words absent from its vocabulary and how GPT tokenizers differ.
  Why it matters: It explains both what the example tokenizer does with unfamiliar words and why that choice does not describe GPT tokenizers.
• Unrelated text boundaries: Explain why <|endoftext|> is placed between independent texts that are concatenated for training.
  Why it matters: A boundary marker preserves the distinction between separate sources even when their text is joined for training.
• Other special token roles: Describe the roles of [BOS], [EOS], and [PAD], and state how the section contrasts them with the simpler GPT token choice.
  Why it matters: These examples show that special tokens can mark different kinds of context, without implying that every tokenizer uses a separate token for each one.

Key terms: SimpleTokenizerV2, <|unk|>, vocabulary, byte pair encoding, subword units, <|endoftext|>, separator token, independent documents, [BOS], [EOS]

2.5 Byte pair encoding

This section introduces BPE tokenization, shows how to use it through tiktoken, and explains why it can represent unfamiliar words.

• Unknown Word Representation: Explain how BPE represents words outside its predefined vocabulary and why it does not need an <|unk|> token for them.
  Why it matters: This is why an unfamiliar word need not be discarded or replaced with an unknown-word token.
• BPE Vocabulary Construction: Describe, at the level given here, how BPE builds vocabulary entries from characters and frequent combinations.
  Why it matters: The construction process explains why BPE has both small pieces and larger word-like tokens available.
• Tiktoken Encoding And Decoding: Describe how the section uses tiktoken to turn text into token IDs and reconstruct text from those IDs, including how to inspect the pieces of an unfamiliar input.
  Why it matters: Encoding and decoding provide a way to see BPE’s token pieces and check that the original input is reconstructed.
• End Of Text Token ID: State the section’s observation about the <|endoftext|> token ID in this BPE vocabulary.
  Why it matters: This gives a concrete example of a special token within the tokenizer’s ID vocabulary.

Key terms: BPE, subword tokens, out-of-vocabulary words, <|unk|>, vocabulary, subwords, frequency cutoff, tiktoken, encode, decode

2.6 Data sampling with a sliding window

This section shows how to turn tokenized text into batches of input–target pairs for next-token training using a sliding window.

• Shifted next-token targets: Explain how a sequence of token IDs becomes input–target pairs for next-token prediction: the targets are the inputs shifted by one position.
  Why it matters: The one-position shift turns ordinary text into the prediction examples used for training.
• Batched input and target tensors: Describe what the dataset and data loader return, and how max_length determines the number of token IDs in each input context and its corresponding targets.
  Why it matters: The loader packages prediction examples into the paired tensors used for training.
• Sliding-window stride and overlap: Explain how stride moves the input window through the token sequence and how its value affects overlap between examples.
  Why it matters: Stride determines which portions of the text become examples and how much they overlap.
• Training batch size tradeoff: Describe the tradeoff the section gives for using small training batches.
  Why it matters: Batch size affects both training memory needs and the noisiness of updates.

Key terms: input–target pairs, input tokens, targets, token IDs, GPTDatasetV1, Dataset, DataLoader, max_length, input tensor, target tensor

2.7 Creating token embeddings

This section shows how an embedding layer turns token IDs into continuous vectors by looking up rows in a trainable weight matrix.

• Embedding purpose and initialization: Explain why GPT-like LLMs use embeddings and how their weights begin before training.
  Why it matters: This explains why token IDs need another representation before they enter the model.
• Embedding matrix row lookup: Describe how an embedding layer uses a token ID to select a vector, including the shapes in the section’s example.
  Why it matters: The row lookup explains exactly how token IDs become the vectors used in later steps.

Key terms: embedding, continuous vector representation, embedding weights, backpropagation, embedding layer, weight matrix, token ID, vocab_size, output_dim, lookup operation

2.8 Encoding word positions

This section explains why LLM inputs need positional information and shows how to add learned absolute positional embeddings to token embeddings.

• Need for positional information: Explain why token embeddings alone do not convey token order to an LLM and why positional information is added.
  Why it matters: Position information lets the model’s input represent where tokens occur, not just which tokens they are.
• Absolute and relative positions: Distinguish absolute positional embeddings from relative positional embeddings as described in this section.
  Why it matters: The distinction clarifies what kind of position information the GPT approach in this section uses.
• GPT input embedding construction: Describe how this section constructs GPT-style input embeddings by adding absolute positional embeddings to token embeddings, including how the example tensor shapes fit together.
  Why it matters: Adding matching positional vectors produces the position-aware input embeddings used by the LLM.

Key terms: token embeddings, self-attention mechanism, position information, absolute positional embeddings, relative positional embeddings, context_length, pos_embeddings, input_embeddings

2.9 Summary

This chapter summary connects tokenization, training-pair creation, and embeddings into a pipeline that prepares text for an LLM.