liveBook · Manning

Chapter 3: A blueprint to modern transformers

What you will learn in chapter 3 of Rearchitecting LLMs

The introduction identifies attention memory growth and MLP compute as distinct bottlenecks that motivate studying transformer components.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction identifies attention memory growth and MLP compute as distinct bottlenecks that motivate studying transformer components.

Key terms: attention mechanism, cache, context, GPU memory, MLP, Multi-Layer Perceptron, parameters, compute time

3.1 Classical architecture: DistilGPT2

This section uses DistilGPT2 to introduce the classical Transformer’s visible structure and relate that structure to the PyTorch classes that define it.

Key terms: DistilGPT2, classic architecture of Transformers, input, Transformer blocks, output, PyTorch model, __init__(), forward(), GPT2Model, GPT2LMHeadModel

3.1.1 General behaviour of a Transformer model

This section traces how DistilGPT2 turns token IDs into input embeddings, processes them through Transformer blocks, and produces scores for the next token.

Key terms: __init__(), forward(), GPT2Model, GPT2Block, h, ln_f, Tokenizer, wte, token embedding, wpe

3.1.2 Classical attention mechanism

This section explains how DistilGPT2 attention contextualizes tokens through multiple QKV attention heads, and why caching their keys and values creates a memory trade-off.

Key terms: input embedding, token embedding, positional embedding, attention mechanism, attention heads, GPT2Attention, c_attn, subspace, Query, Key

3.1.3 The classic MLP mechanism

Explains how DistilGPT2’s classic MLP follows attention, transforms contextualized embeddings through expansion and nonlinear activation, and motivates width pruning through neuron specialization.

Key terms: Transformer block, attention module, MLP module, contextual information, forward(), __init__(), c_fc, act, c_proj, dropout

3.1.4 The Transformer dimensions: depth and width

This section defines transformer depth and width, connects each dimension to pruning, and compares wide models with deep, narrow ones.

Key terms: depth, Transformer blocks, depth pruning, latency, width, intermediate dimension, MLP block, width pruning, expansion capacity, wide models

3.2 The modern Transformer architecture

Introduces the scaling pressures behind modern transformer designs and compares Llama-3.2-1B with DistilGPT2 to distinguish greater scale from architectural changes.

Key terms: Attention, MLP, Grouped-Query Attention (GQA), Gated Linear Units (GLU), vocabulary, width, depth, Transformer blocks, rotary_emb, LlamaRotaryEmbedding

3.2.1 Optimized attention: from Multi-Head Attention (MHA) to Grouped-Query Attention (GQA)

The section shows how Llama-3.2’s grouped-query attention differs structurally from classic multi-head attention, saves memory, and complicates individual attention-head pruning.

Key terms: DistilGPT2, Llama-3.2, Q, K, V, o_proj, GQA, MHA, Queries, Keys

3.2.2 The evolution of the MLP: from simple expansion to Gated Linear Units (GLU)

This section contrasts classic MLP filtering with GLU’s input-dependent gate, traces the GLU processing flow, and explains what that structure means for width pruning.

Key terms: GELU, Gated Linear Units (GLU), learned multiplicative gate, up_proj, gate_proj, SiLU, element-wise multiplication, down_proj, width pruning, expansion ratio

3.3 Connecting structure, behavior, and optimization

The section connects a transformer’s static structure to its input-dependent activations and explains why observing those activations can guide targeted optimization and behavioral correction.

Key terms: weights, activations, activation vector, structure, behavior, surgical interventions

3.4 Hands-on lab

3.5 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: An end-to-end rearchitecting project Chapter 4: Building smaller and faster LLMs with depth pruning →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.