A transformer block is a repeated processing unit that transforms representations using attention and a feed-forward MLP. Attention contextualizes information; the MLP processes it. Blocks also contain normalization layers and may include dropout. section 3.5 section 4.1.1
A block is the complete repeated unit; a module is a functional component such as attention or the MLP; and a layer is an individual operation such as a linear projection or normalization. Code may use “layers” for the block collection, as in model.model.layers. section 4.1
Inspecting a block stack
DistilGPT2 has six GPT2Block instances between its input and output components. Its model forward pass calls each block in sequence; each block calls its attention module and then its MLP. The block diagram also shows normalization before each module. section 3.1 section 3.1.1
Figure 3.2 Anatomy of DistilGPT2. On the left, the general architecture is shown: data flows from the Embeddings layer, passes through the stack of six Transformer Blocks, and ends at the output layer. On the right, a "zoom" into one of those blocks reveals a LayerNorm → GPT2Attention → LayerNorm → GPT2MLP flow, where each module processes the information before passing it to the next.
Trace the stack of six blocks and the attention and MLP flow inside one block. section 3.1.1
Depth is the number of stacked blocks; width is the MLP's intermediate dimension. DistilGPT2 has six blocks and an MLP width of 3,072. More depth permits more sequential refinement but generally adds latency; reducing width affects parameter count and memory. section 3.1.4
Depth pruning removes whole blocks, including transformations learned during training. It can reduce latency and memory use by removing parameters, memory-access sequences, and the removed attention modules' contributions to the KV cache. The change is aggressive, and which blocks are removed affects model performance. section 4.1.1 section 4.1.2
Inspect before modifying
For Gemma-3-270M, count parameters and inspect model.model.layers before changing depth. The reported model has about 268 million parameters and 18 Gemma3DecoderLayer blocks. Establish a generation sanity check and benchmark baseline so changes can be compared against the original. section 2.2
original_params= sum(p.numel() for p in model.parameters())
print(f"Total parameters: {original_params}")
print(f"Original layers: {len(model.model.layers)}")
print("=" * 20)
print(model)
Use this listing to count parameters and blocks and print the model structure. section 2.2




