Chapter 3: A blueprint to modern transformers
The introduction identifies attention memory growth and MLP compute as distinct bottlenecks that motivate studying transformer components.
After this chapter you will be able to
- Attention cache memory growth. Explain why inference memory use can exceed what the model size alone suggests: attention creates a context cache that grows with each processed token and can exhaust GPU memory.
- MLP parameter and compute cost. Describe the MLP as the block responsible for processing knowledge and explain that it consumes most of the model's parameters and compute time.
- Static structure and model code. Describe the three broad parts of the printed DistilGPT2 structure—input, Transformer blocks, and output—and explain why that static map is paired with inspection of the PyTorch class. State that `__init__()` declares and initializes layers, while `forward()` is the other fundamental method named as defining model behavior.
- Nested forward execution. Trace how forward() methods determine execution order: GPT2Model passes embeddings through its blocks and final normalization, while each GPT2Block calls its attention and MLP submodules.
- Input embeddings and dropout. Describe how external tokenization supplies IDs to wte, how wpe supplies position vectors, and why their vectors can be added to form input embeddings. State what dropout does during training versus inference.
- Transformer block responsibilities. Describe the ordered components of a DistilGPT2 block and distinguish attention's contextualization from the MLP's processing of that contextualized information.
- Final token prediction. Trace how processed block outputs reach final normalization and the language-modeling head, and explain why the immediate output is vocabulary-wide scores rather than a selected token.
- Contextualizing token embeddings. Explain why a token’s identity and position do not establish its meaning in context, and how attention enriches its embedding with information from other tokens.
- Parallel attention heads. Describe how DistilGPT2’s attention heads process an embedding in parallel, how their different patterns arise, and how their results are brought together.
- Query key value weighting. Trace how a token’s Query, other tokens’ Keys, and their Values determine the information mixed into that token’s embedding.
- KV cache trade-offs. Explain why generation uses a KV Cache, how its memory grows, and how MQA and GQA reduce that pressure compared with MHA.
- Attention and MLP roles. Explain the order of attention and MLP within each DistilGPT2 block and how their roles differ in the section’s example.
- MLP execution and dimensions. Trace the classic MLP’s actual processing order and describe how the embedding dimensions change.
- Purpose of nonlinear activation. Justify why an activation is placed between expansion and contraction and describe what GELU does to neuron signals.
- Transformer depth and depth pruning. Explain that depth is the number of stacked Transformer blocks and that depth pruning removes entire blocks. Describe the trade-off between sequential refinement and latency.
- MLP width and width pruning. Explain that width here refers specifically to the MLP block’s intermediate dimension, and describe how reducing its expansion capacity affects parameters and memory.
- Modern block design motivations. Explain why the classical attention and MLP modules prompted changes as models scaled, and identify the role attributed here to GQA and GLU.
- Grouped queries and memory. Explain how grouped-query attention shares keys and values while retaining distinct queries, and relate that sharing to the section’s memory comparison.
- Grouped attention pruning risk. Explain why removing an individual attention head is more straightforward in classic MHA than in GQA.
- Static versus adaptive filtering. Explain why the classic MLP’s fixed activation is limited as a filter and how GLU adds input-dependent control.
- Classic and GLU processing flows. Trace the classic MLP’s expansion–activation–contraction flow and the GLU flow through up_proj, gate_proj, element-wise multiplication, and down_proj.
- Paired neurons in width pruning. Explain why width pruning in a GLU MLP must treat corresponding up_proj and gate_proj neurons as pairs.
- Weights and activations. Distinguish learned weights from activations, and describe how a layer produces an activation vector for a specific input.
- Structure behavior and intervention. Explain how a model’s structure differs from its observed behavior, and why examining activations can help locate parts associated with unwanted behavior or targets for optimization.
What it builds on
- Applying depth pruning (chapter 2) · Introduced depth pruning by removing blocks.
Section by section
Introduction
The introduction identifies attention memory growth and MLP compute as distinct bottlenecks that motivate studying transformer components.
- Attention cache memory growth: Explain why inference memory use can exceed what the model size alone suggests: attention creates a context cache that grows with each processed token and can exhaust GPU memory.
Why it matters: This explains why a model that fits in memory may still run out of memory on a long document. - MLP parameter and compute cost: Describe the MLP as the block responsible for processing knowledge and explain that it consumes most of the model's parameters and compute time.
Why it matters: Locating the MLP's costs helps distinguish compute and parameter concerns from attention's context-memory concern.
Key terms: attention mechanism, cache, context, GPU memory, MLP, Multi-Layer Perceptron, parameters, compute time
3.1 Classical architecture: DistilGPT2
This section uses DistilGPT2 to introduce the classical Transformer’s visible structure and relate that structure to the PyTorch classes that define it.
- DistilGPT2 as classical example: Explain why DistilGPT2 is used to introduce the classical Transformer architecture: it is a relatively small, 82-million-parameter model that retains the fundamental structure without the modern evolutions discussed later.
Why it matters: This explains why the book starts with DistilGPT2 rather than a more modern model. - Static structure and model code: Describe the three broad parts of the printed DistilGPT2 structure—input, Transformer blocks, and output—and explain why that static map is paired with inspection of the PyTorch class. State that `__init__()` declares and initializes layers, while `forward()` is the other fundamental method named as defining model behavior.
Why it matters: It connects the component listing to the code that defines the model without treating the listing as a complete account of its behavior. - Base model and head wrapper: Distinguish the inspected `GPT2Model` constructor from `GPT2LMHeadModel`: explain why `lm_head` appears in the printed structure but not in that constructor.
Why it matters: This resolves an apparent mismatch between the printed model and the constructor being inspected.
Key terms: DistilGPT2, classic architecture of Transformers, input, Transformer blocks, output, PyTorch model, __init__(), forward(), GPT2Model, GPT2LMHeadModel
3.1.1 General behaviour of a Transformer model
This section traces how DistilGPT2 turns token IDs into input embeddings, processes them through Transformer blocks, and produces scores for the next token.
- Nested forward execution: Trace how forward() methods determine execution order: GPT2Model passes embeddings through its blocks and final normalization, while each GPT2Block calls its attention and MLP submodules.
Why it matters: Reading forward() lets you trace what the model actually does, rather than inferring execution order from its component definitions. - Input embeddings and dropout: Describe how external tokenization supplies IDs to wte, how wpe supplies position vectors, and why their vectors can be added to form input embeddings. State what dropout does during training versus inference.
Why it matters: The blocks receive vectors that encode both token identity and sequence position, not raw text. - Transformer block responsibilities: Describe the ordered components of a DistilGPT2 block and distinguish attention's contextualization from the MLP's processing of that contextualized information.
Why it matters: This division between contextualization and knowledge processing is the section's basic map of what happens inside each block. - Final token prediction: Trace how processed block outputs reach final normalization and the language-modeling head, and explain why the immediate output is vocabulary-wide scores rather than a selected token.
Why it matters: It connects the block's internal representations to the model's next-token prediction. - Tokens versus words: Distinguish the section's simplified one-token-per-word examples from its statement that actual token-to-word conversion varies.
Why it matters: It keeps the word-based examples from becoming an incorrect rule about model inputs.
Key terms: __init__(), forward(), GPT2Model, GPT2Block, h, ln_f, Tokenizer, wte, token embedding, wpe
3.1.2 Classical attention mechanism
This section explains how DistilGPT2 attention contextualizes tokens through multiple QKV attention heads, and why caching their keys and values creates a memory trade-off.
- Contextualizing token embeddings: Explain why a token’s identity and position do not establish its meaning in context, and how attention enriches its embedding with information from other tokens.
Why it matters: It explains what the attention layer adds to the representation of each token. - Parallel attention heads: Describe how DistilGPT2’s attention heads process an embedding in parallel, how their different patterns arise, and how their results are brought together.
Why it matters: The heads explain how attention can examine different contextual relationships at once. - Query key value weighting: Trace how a token’s Query, other tokens’ Keys, and their Values determine the information mixed into that token’s embedding.
Why it matters: QKV weighting is the mechanism by which attention selects contextual information. - Classical MHA projections: Describe how DistilGPT2’s Multi-Head Attention generates QKV projections for its heads.
Why it matters: It establishes what is replicated across heads in the classical architecture. - KV cache trade-offs: Explain why generation uses a KV Cache, how its memory grows, and how MQA and GQA reduce that pressure compared with MHA.
Why it matters: It explains both the memory bottleneck of classical attention and the trade-off behind alternative head arrangements.
Key terms: input embedding, token embedding, positional embedding, attention mechanism, attention heads, GPT2Attention, c_attn, subspace, Query, Key
3.1.3 The classic MLP mechanism
Explains how DistilGPT2’s classic MLP follows attention, transforms contextualized embeddings through expansion and nonlinear activation, and motivates width pruning through neuron specialization.
- Attention and MLP roles: Explain the order of attention and MLP within each DistilGPT2 block and how their roles differ in the section’s example.
Why it matters: This distinguishes what the MLP adds to the contextual information it receives. - MLP execution and dimensions: Trace the classic MLP’s actual processing order and describe how the embedding dimensions change.
Why it matters: The execution order shows where the MLP gains processing capacity and where its activation acts. - Purpose of nonlinear activation: Justify why an activation is placed between expansion and contraction and describe what GELU does to neuron signals.
Why it matters: Nonlinearity explains why widening and narrowing the embedding is more useful than stacking two linear projections. - Neuron specialization and pruning: Explain how specialized MLP neurons motivate selectively removing neurons for a narrower model.
Why it matters: Neuron specialization connects the MLP’s structure to the possibility of making a focused model smaller.
Key terms: Transformer block, attention module, MLP module, contextual information, forward(), __init__(), c_fc, act, c_proj, dropout
3.1.4 The Transformer dimensions: depth and width
This section defines transformer depth and width, connects each dimension to pruning, and compares wide models with deep, narrow ones.
- Transformer depth and depth pruning: Explain that depth is the number of stacked Transformer blocks and that depth pruning removes entire blocks. Describe the trade-off between sequential refinement and latency.
Why it matters: Depth tells you what is removed in depth pruning and why changing it can affect both processing and latency. - MLP width and width pruning: Explain that width here refers specifically to the MLP block’s intermediate dimension, and describe how reducing its expansion capacity affects parameters and memory.
Why it matters: Width identifies a different structural choice from removing blocks, with a direct bearing on model size and memory. - Wide versus deep model shapes: Distinguish the wide and deep-narrow model shapes described here using their embedding sizes, MLP expansion, and emphasis on per-layer processing versus sequential transformations.
Why it matters: The comparison shows how architects can distribute model capacity across width and depth in different ways.
Key terms: depth, Transformer blocks, depth pruning, latency, width, intermediate dimension, MLP block, width pruning, expansion capacity, wide models
3.2 The modern Transformer architecture
Introduces the scaling pressures behind modern transformer designs and compares Llama-3.2-1B with DistilGPT2 to distinguish greater scale from architectural changes.
- Modern block design motivations: Explain why the classical attention and MLP modules prompted changes as models scaled, and identify the role attributed here to GQA and GLU.
Why it matters: This connects the modern block components to the problems they are meant to address. - Scale versus architectural changes: Distinguish Llama-3.2-1B's larger vocabulary, width, and depth from the changes the section identifies as the more significant architectural evolution.
Why it matters: A larger model is not necessarily a differently designed model. - Positional and normalization replacements: Describe what Llama's rotary embedding and RMSNorm replace in DistilGPT2, including the benefit the section attributes to each.
Why it matters: These replacements show that the models differ in how they represent position and normalize computations, not just in size.
Key terms: Attention, MLP, Grouped-Query Attention (GQA), Gated Linear Units (GLU), vocabulary, width, depth, Transformer blocks, rotary_emb, LlamaRotaryEmbedding
3.2.1 Optimized attention: from Multi-Head Attention (MHA) to Grouped-Query Attention (GQA)
The section shows how Llama-3.2’s grouped-query attention differs structurally from classic multi-head attention, saves memory, and complicates individual attention-head pruning.
- Attention projection structure: Describe how DistilGPT2 and Llama-3.2 organize their attention projections, including the role of Llama’s output projection.
Why it matters: The projection layout makes the architectural change visible when you inspect a model. - Grouped queries and memory: Explain how grouped-query attention shares keys and values while retaining distinct queries, and relate that sharing to the section’s memory comparison.
Why it matters: Sharing keys and values explains both Llama’s smaller K/V projections and the memory advantage illustrated here. - Grouped attention pruning risk: Explain why removing an individual attention head is more straightforward in classic MHA than in GQA.
Why it matters: An optimization that is simple for independent MHA heads may not be safe for grouped heads.
Key terms: DistilGPT2, Llama-3.2, Q, K, V, o_proj, GQA, MHA, Queries, Keys
3.2.2 The evolution of the MLP: from simple expansion to Gated Linear Units (GLU)
This section contrasts classic MLP filtering with GLU’s input-dependent gate, traces the GLU processing flow, and explains what that structure means for width pruning.
- Static versus adaptive filtering: Explain why the classic MLP’s fixed activation is limited as a filter and how GLU adds input-dependent control.
Why it matters: The distinction explains why GLU changes how an MLP filters information, not merely how many layers it has. - Classic and GLU processing flows: Trace the classic MLP’s expansion–activation–contraction flow and the GLU flow through up_proj, gate_proj, element-wise multiplication, and down_proj.
Why it matters: Following both paths shows where GLU’s contextual filtering enters the MLP. - Paired neurons in width pruning: Explain why width pruning in a GLU MLP must treat corresponding up_proj and gate_proj neurons as pairs.
Why it matters: The paired structure is a constraint on reducing a GLU model’s width. - Model-dependent expansion trade-offs: Describe what the section’s pruning experiments suggest about lowering the expansion ratio and why the result cannot be assumed for every model.
Why it matters: The reported trade-off is a reason to measure pruning outcomes rather than assume one expansion ratio suits every model.
Key terms: GELU, Gated Linear Units (GLU), learned multiplicative gate, up_proj, gate_proj, SiLU, element-wise multiplication, down_proj, width pruning, expansion ratio
3.3 Connecting structure, behavior, and optimization
The section connects a transformer’s static structure to its input-dependent activations and explains why observing those activations can guide targeted optimization and behavioral correction.
- Weights and activations: Distinguish learned weights from activations, and describe how a layer produces an activation vector for a specific input.
Why it matters: This distinction makes it possible to talk about what a model does for an input, not just what parameters it contains. - Structure behavior and intervention: Explain how a model’s structure differs from its observed behavior, and why examining activations can help locate parts associated with unwanted behavior or targets for optimization.
Why it matters: Looking at both anatomy and input-dependent behavior helps identify where a targeted change might matter.
Key terms: weights, activations, activation vector, structure, behavior, surgical interventions
3.4 Hands-on lab
3.5 Summary
Where this is used later
- What the KV cache is and why it matters (chapter 8) · Introduces the cache whose memory cost motivates attention optimization.
- Selecting static neurons (chapter 5) · Focuses on selecting neurons in the MLP for width pruning.
- How depth pruning works (chapter 4) · The general behavior of a Transformer model.
- Using PyTorch hooks (chapter 4) · The model processes data through transformer blocks.
- The foundations of LoRA (chapter 7) · Introduces the attention module whose q_proj weight matrix is used as the example.
- Dynamic inference techniques (chapter 9) · Assumes familiarity with the MLP module that experts replace.
- Identifying neurons for removal (chapter 5) · Motivates identifying neurons for removal during width pruning.
- Fundamentals and benefits of depth pruning (chapter 4) · Introduces transformer depth as an architectural dimension.
Concepts explained in this chapter
- multilayer perceptronA multilayer perceptron is a feed-forward neural network composed of successive learned linear transformations and nonlinear activations.
- grouped-query attentionGrouped query attention is an attention mechanism in which multiple query heads share key and value heads.
- gated linear unitA gated linear unit is a neural-network structure that modulates one linear transformation with a learned gate derived from another transformation.
- attentionAttention is a neural-network mechanism that computes input-dependent weights to prioritize information during processing.
- attention headAn attention head is an independent attention mechanism that learns to focus on particular types of relationships among sequence elements.
- keyA key vector is an attention representation matched against queries to determine attention weights.
- multi-head attentionMulti-head attention is an attention mechanism that computes attention through multiple parallel representation subspaces and combines their outputs.
- queryA query vector is an attention representation used to determine how strongly a position should attend to available keys.
- width pruningWidth pruning is a structured pruning technique that removes selected neurons or channels to reduce a model's width.
- key-value cacheA key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.
- transformer blockA transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.
- inner productA dot product is a scalar operation between two equal-length vectors that measures their directional alignment.
- multi-query attentionMulti-query attention is an attention architecture in which multiple query heads share a single set of key and value vectors.
- qkv mechanismThe QKV mechanism computes query, key, and value vectors so attention can score relationships and combine contextual information.
- softmaxSoftmax converts a vector of scores into nonnegative probabilities that sum to one.
- valueA value is a vector containing the semantic information that attention aggregates from tokens selected by query-key compatibility.
- GELUGELU is a smooth neural-network activation function that weights inputs using the Gaussian error function.
- GPT-2GPT2Model is a GPT-2 transformer model implementation that organizes embeddings, transformer blocks, normalization, and output projection.
- forward methodA forward method is a neural-module method that defines the computation performed when input data passes through the module.
- language modeling headA language modeling head maps model representations to scores over a language model's vocabulary.
- expansion ratioAn expansion rate is the ratio of a module's intermediate dimension to the model's hidden-state dimension, expressed as a percentage or factor.
- input embeddingAn input embedding is a vector representation of a token that combines information about the token and its position before contextual processing.
- positional embeddingA positional embedding is a learned or fixed vector representation that supplies sequence-position information to a neural language model.
- token embeddingA token embedding is a vector representation encoding the identity or learned meaning of a token independently of its surrounding sequence context.
- transformer architectureA transformer is a neural-network architecture that processes representations using attention and position-wise feed-forward components.
- content projectionA content projection is the linear transformation in a gated unit that produces the representation modulated by the gate.
- down projection layerA down projection is a linear transformation that reduces an intermediate representation back to the model's hidden dimension.
- element-wise multiplicationElement-wise multiplication is the operation of multiplying corresponding entries of two equally sized vectors or arrays.
- gate projectionA gate projection produces the signal that controls another intermediate signal in a gated feed-forward block.
- multiplicative gateA multiplicative gate is a learned vector that scales another representation element by element to control information flow.
- activationAn activation is a value produced by a neural-model unit during computation.
- SwishSiLU is a smooth activation function defined by multiplying an input by its sigmoid.
- depth pruningDepth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- valueA value vector is the attention representation combined according to attention weights to produce an attended output.
- neural network layerA model layer is a computational stage in a neural network that transforms incoming representations.
- optimizationAn optimization technique is a method for improving a model's efficiency or task performance through changes to its design, parameters, or configuration.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
