liveBook · Manning

Block selection strategy

As explained in Rearchitecting LLMs

A block selection strategy is a method for choosing which transformer blocks to retain or remove during depth pruning.

Block selection strategy chooses which complete Transformer blocks to remove during depth pruning. Selection matters: removing the same number of blocks can produce the same inference gain but very different performance costs. section 4.1 section 4.1.2

Compared approaches include:

  • Data-driven individual selection: remove the least-contributing blocks identified through cosine-similarity importance analysis, whether consecutive or not. section 6.1.1
  • Data-driven consecutive selection: apply the same importance analysis while requiring the removed blocks to be consecutive. section 6.1.1
  • Heuristic strategies provide alternatives to data-driven importance analysis. section 6.1.1

Initial comparison

The experiment removes four of Gemma 3 270M’s 18 Transformer blocks and first evaluates strategies on 2,000 Cosmopedia records before scaling the winning approach. section 6.1.1

Figure 4.2 Impact of block selection on depth pruning. The gain in inference (+15%) is identical as we are removing four blocks in both scenarios. The performance cost varies drastically depending on which ones are removed.

Compare how removing the same number of blocks can yield different performance costs. section 4.1.2

Figure 6.1 Comparison of layer removal strategies. All experiments remove 4 blocks (striped) from the original 18-block model. EXP01-02 use data-driven importance analysis (Listing 6.5), while EXP03-04 follow heuristic rules.

View the four-block removal strategies compared on the 18-block model. section 6.1.1

How Rearchitecting LLMs covers it

How deep it goes
Compares four strategies for removing four of 18 blocks, including individual or consecutive importance-based selection and heuristic rules.
Examples
Uses Gemma 3 270M, four-block removal, 2,000 Cosmopedia records, and individual-versus-consecutive importance-based selection.
What it assumes
Assumes depth-pruning fundamentals and defines a block as an Attention-plus-MLP unit.
What you can do after
Evaluate four block-removal strategies on a 2,000-record sample before scaling the winning approach.
How it connects
Connects depth pruning to cosine-similarity importance analysis and subsequent knowledge recovery.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.