liveBook · Manning

Transformer block

As explained in Rearchitecting LLMs

A transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.

A transformer block is a repeated processing unit that transforms representations using attention and a feed-forward MLP. Attention contextualizes information; the MLP processes it. Blocks also contain normalization layers and may include dropout. section 3.5 section 4.1.1

A block is the complete repeated unit; a module is a functional component such as attention or the MLP; and a layer is an individual operation such as a linear projection or normalization. Code may use “layers” for the block collection, as in model.model.layers. section 4.1

Inspecting a block stack

DistilGPT2 has six GPT2Block instances between its input and output components. Its model forward pass calls each block in sequence; each block calls its attention module and then its MLP. The block diagram also shows normalization before each module. section 3.1 section 3.1.1

Figure 3.2 Anatomy of DistilGPT2. On the left, the general architecture is shown: data flows from the Embeddings layer, passes through the stack of six Transformer Blocks, and ends at the output layer. On the right, a "zoom" into one of those blocks reveals a LayerNorm → GPT2Attention → LayerNorm → GPT2MLP flow, where each module processes the information before passing it to the next.

Trace the stack of six blocks and the attention and MLP flow inside one block. section 3.1.1

Depth is the number of stacked blocks; width is the MLP's intermediate dimension. DistilGPT2 has six blocks and an MLP width of 3,072. More depth permits more sequential refinement but generally adds latency; reducing width affects parameter count and memory. section 3.1.4

Depth pruning removes whole blocks, including transformations learned during training. It can reduce latency and memory use by removing parameters, memory-access sequences, and the removed attention modules' contributions to the KV cache. The change is aggressive, and which blocks are removed affects model performance. section 4.1.1 section 4.1.2

Inspect before modifying

For Gemma-3-270M, count parameters and inspect model.model.layers before changing depth. The reported model has about 268 million parameters and 18 Gemma3DecoderLayer blocks. Establish a generation sanity check and benchmark baseline so changes can be compared against the original. section 2.2

original_params= sum(p.numel() for p in model.parameters())
print(f"Total parameters: {original_params}")
print(f"Original layers: {len(model.model.layers)}")
print("=" * 20)
print(model)

Use this listing to count parameters and blocks and print the model structure. section 2.2

How Rearchitecting LLMs covers it

How deep it goes
Traces nested forward calls and block components, then explains the effects of removing entire blocks.
Examples
Inspects DistilGPT2's six-block stack and Gemma-3-270M's 18 decoder blocks.
What you can do after
Evaluate model structure and pruning trade-offs by inspecting block counts, components, and baseline behavior.
How it connects
Connects attention and MLP modules to normalization, depth pruning, width pruning, and the KV cache.

Also explained in

Compare how 5 books explain transformer block →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.