Chapter 2: Generating text with a pretrained LLM

Introduction

The chapter establishes why it starts with a pretrained base LLM before adding reasoning techniques in later chapters.

• Base model as baseline: Explain why the chapter starts with a pretrained base LLM rather than an instruction- or preference-tuned assistant: it makes capabilities contributed by later reasoning methods easier to identify.
  Why it matters: The base model provides a clear starting point for seeing what later reasoning methods add.

Key terms: conventional LLM, pretrained base model, nonreasoning LLM, reasoning methods

2.1 Introducing LLMs for text generation

This section introduces the pretrained base model used for text generation and explains how to read the chapter’s LLM diagrams.

• Pretrained Base Model Capabilities: Explain that the chapter uses a base model that has undergone only pretraining, yet can generate coherent text and sometimes follow basic instructions.
  Why it matters: This sets expectations for what the model can do before any additional training.
• Neural Network Diagram Direction: Describe the convention for reading LLM diagrams: inputs appear at the bottom, outputs at the top, and arrows show information flowing upward through the model.
  Why it matters: This convention helps the reader interpret the chapter’s model diagrams.

Key terms: base model, pretraining, coherent text, basic instructions, inputs, outputs, flow of information

2.2 Setting up the coding environment

This section explains how to prepare a Python environment and run the book’s code using either pip or uv.

• Pip dependency setup: Describe when the pip-based setup is appropriate and how the book’s dependencies can be installed.
  Why it matters: This gives readers with an existing Python installation a straightforward way to prepare for the examples.
• Uv workflow advantages: Explain why the author recommends uv as an alternative to pip and what environment work uv handles.
  Why it matters: Uv can reduce the manual environment and dependency setup needed to run the book’s code.
• Repository and code launch: Describe the main steps for obtaining the repository and starting its notebooks or Python scripts with uv.
  Why it matters: Obtaining and opening the project lets readers execute the chapter’s examples.

Key terms: Python 3.10, pip, requirements.txt, PyTorch, uv, dependency resolution, isolated environments, pyproject.toml, .venv/, GitHub repo

2.3 Understanding hardware needs and recommendations

The section explains why the book uses a smaller pretrained LLM and how readers can choose between CPU, local GPU, and cloud resources for its exercises.

• Smaller pretrained model rationale: Justify using a smaller pretrained LLM rather than training a new base model, and explain why it can still be useful for learning reasoning techniques.
  Why it matters: You can practice the book's reasoning techniques without paying to train a large model from scratch.
• CPU and GPU needs: Distinguish what can run reasonably on a CPU from the later work that benefits from a GPU, and describe PyTorch's role in checking hardware and running the book's models.
  Why it matters: You do not need to postpone the early exercises just because your computer lacks a GPU.
• Cloud GPU alternatives: Describe the cloud option for readers without suitable local GPU hardware and give the examples named in the section.
  Why it matters: Lacking a local GPU does not rule out trying GPU-accelerated work later.

Key terms: base-model LLM, pretrained LLM, reasoning techniques, CPU, GPU, PyTorch, PyTorch-supported GPU, cloud resources, Lightning AI Studio, Google Colab

2.4 Preparing input texts for LLMs

This section shows how a tokenizer converts text to and from token IDs, then explains subword tokenization and the tradeoff between vocabulary size and sequence length.

• Token ID round trip: Explain why LLM text must be represented as token IDs and describe how encoding a prompt into IDs and decoding those IDs can reconstruct the prompt.
  Why it matters: Token IDs are the form of text the LLM can process.
• Subword token segmentation: Describe how the tokenizer can represent text using whole-word or subword tokens, including how spaces may be part of tokens.
  Why it matters: A token is not necessarily a whole word, which affects how many IDs a prompt contains.
• Vocabulary size tradeoff: Explain the tradeoff between a larger token vocabulary and the length of tokenized sequences.
  Why it matters: Both vocabulary size and the number of tokens in a sequence affect generation costs.

Key terms: tokenizer, token IDs, encode, decode, tokenization round trip, byte pair encoding (BPE), subword, token, word boundaries, vocabulary

2.5 Loading pretrained models

The section explains why the book uses Qwen3 0.6B and how to download its pretrained weights, load them into a local PyTorch model, and place the model on a device.

• Qwen3 Base Model Choice: Explain why the book uses Qwen3 0.6B as its pretrained base model.
  Why it matters: The model choice makes the book’s experiments practical while providing a reasoning-model comparison point.
• Local PyTorch Model Implementation: Describe how the book’s Qwen3 implementation relates to the official weights and where it runs.
  Why it matters: This setup lets readers modify the model code while keeping inputs and outputs on their own machine.
• CPU And GPU Selection: Describe how the section chooses a device and why a reader might temporarily choose the CPU.
  Why it matters: Starting on the CPU can simplify troubleshooting before using a faster available GPU.
• Pretrained Weight Loading Sequence: Trace the steps that turn the downloaded Qwen3 weights into a loaded model.
  Why it matters: Downloading a tokenizer alone does not initialize the pretrained model needed for generation.
• Model Architecture At A Glance: Describe what the printed model summary shows and how much architecture detail this book requires.
  Why it matters: The summary confirms the model’s broad structure without requiring readers to master its internal architecture.

Key terms: Qwen3 0.6B, open-weight model, base model, reasoning variant, custom reimplementation, pure PyTorch, pretrained weights, locally, device, CPU

2.6 Understanding the sequential LLM text generation process

The section explains how an LLM generates text one token at a time by selecting and decoding a next token from its output scores.

• Autoregressive token generation: Explain that LLM text generation is sequential: the model generates one token at a time, then uses the growing sequence to generate the next token.
  Why it matters: This is the basic process the later text-generation function will repeat.
• Positionwise output scores: Describe what the model returns for an input sequence and why generation uses only the last position's prediction.
  Why it matters: Selecting the right row of model output is necessary to generate the next unseen token.
• Highest score token decoding: Trace how the last position's scores become a text token, distinguishing the highest score from the index that identifies its token.
  Why it matters: The model's numeric output must be turned into a token ID and then readable text.
• Inference mode efficiency: Explain why text generation runs under PyTorch inference mode rather than tracking gradients.
  Why it matters: Avoiding unnecessary gradient tracking makes generation more efficient.
• Singleton tensor dimensions: Describe how squeeze, unsqueeze, and keepdim affect tensor shapes in the examples.
  Why it matters: Maintaining compatible tensor shapes helps the selected token work with the tokenizer and later concatenation.

Key terms: autoregressive text generation, token, next-token prediction, forward pass, input position, vocabulary size, last position, score matrix, argmax, max

2.7 Coding a minimal text generation function

The section implements basic sequential text generation with a pretrained LLM, adds end-of-sequence stopping and speed measurement, and clarifies what inference means in this context.

• Sequential token generation function: Trace how generate_text_basic_stream turns a prompt into generated text: tokenize the prompt, repeatedly select and append the next token, then decode the generated token IDs.
  Why it matters: This is the basic generation process that the later improvements build on.
• End-of-sequence stopping: Explain why generation should stop at <|endoftext|> and how its token ID is supplied to the generation function.
  Why it matters: Stopping at the end-of-sequence token prevents the model from continuing past its intended response.
• Generation performance measurements: Describe what generate_stats measures and how the generation run is timed.
  Why it matters: These measurements make the basic function's slowness observable before speed improvements are introduced.
• Neural network inference meaning: Distinguish neural network inference during text generation from statistical inference.
  Why it matters: It clarifies that generating a token applies an already trained model rather than training it.

Key terms: generate_text_basic_stream, argmax, token IDs, max_new_tokens, <|endoftext|>, tokenizer.eos_token_id, end-of-sequence token, generate_stats, tokens per second, runtime

2.8 Faster inference via KV caching

This section explains how KV caching avoids repeated work during text generation, shows how to use it in the generation function, and compares its speed with the uncached version.

• Redundant work in generation: Explain why feeding the entire growing sequence back into the model at every step is inefficient and how KV caching avoids that repeated work.
  Why it matters: This is the reason caching can speed up generation without changing the prompt or generated tokens.
• Cached generation function flow: Trace what the cached generation function passes to the model on its first iteration and on subsequent iterations.
  Why it matters: The change in what is passed to the model makes the cache’s role concrete in the code.
• Decoding work and observed speed: Describe the rough change in decoding work with KV caching and what the section’s timing and output comparison demonstrate.
  Why it matters: The work estimate explains the speedup, while matching output checks that the optimization has not changed the result.

Key terms: KV caching, keys and values, attention mechanism, intermediate representations, generate_text_basic_stream_cache, KVCache, token_ids, next_token, O(n2), O(n)

2.9 Faster inference via PyTorch model compilation

This section explains how PyTorch compilation can speed up text generation, compares its measured effect with and without KV caching, and explains limits of the hardware benchmarks.

• PyTorch compilation speedup: Explain how torch.compile can make repeated text generation faster by optimizing groups of model operations and reducing execution overhead.
  Why it matters: Even a modest per-step speedup can save time when generation runs repeatedly.
• Compiled generation performance comparisons: Describe why the first compiled run is a poor speed measurement and compare the reported speedups with and without KV caching.
  Why it matters: The first run and the choice of cached or noncached generation can give very different impressions of compilation’s benefit.
• KV cache memory tradeoff: Explain why this implementation grows its KV cache on demand rather than preallocating it for faster GPU execution.
  Why it matters: The cache design helps explain why these GPU results do not show the largest possible speed gains.
• Hardware benchmark limitations: Explain why the reported NVIDIA GPU advantage is not larger for this particular benchmark.
  Why it matters: Benchmark results depend on the model, implementation, and workload—not just the hardware.

Key terms: torch.compile, computation graph, optimized kernels, Python overhead, initial compilation, tokens per second, KV cache, noncached, K and V tensors, preallocation

2.10 Summary

Summarizes the components of a pretrained-LLM text generation pipeline, its token-by-token operation, and two ways to improve generation speed.