liveBook · Manning

Adjacent block protection

As explained in Rearchitecting LLMs

Adjacent block protection is a pruning constraint that prevents or limits the removal of neighboring model blocks.

Adjacent block protection prevents a depth-pruning selector from removing consecutive transformer blocks. Applied during importance-ranked selection, it skips a candidate if a neighboring block has already been selected. It can be enabled independently of heuristic protection for early and late blocks. section 4.3.4

WinoGrande block selection

For four removals, the unprotected selection is [25, 26, 24, 23]. With adjacent and heuristic protection both enabled, it is [25, 23, 13, 15]. The comparison changes the selection strategy, not the number of blocks removed; benchmark evaluation is needed to assess the effect. section 4.3.4

Listing 4.8 Select blocks to remove.
def select_layers_to_prune(importance_scores,
                           num_layers=2,
                           heuristic_protection=True,
                           adjacent_protection=True):
 
    sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])
 
    selected = []
    for layer, score in sorted_layers:
        if heuristic_protection and layer in [0, 1, 2, 3, 26, 27]:  
            continue
 
        if adjacent_protection and any(abs(layer - l) == 1 for l in selected):
            continue
 
        selected.append(layer)
 
        if len(selected) >= num_layers:
            break
 
    return selected

The selector shows how adjacency protection skips a candidate next to a previously selected block. section 4.3.4

Test protection choices by:

  • Disabling heuristic and adjacent protection separately, then evaluating each resulting model on WinoGrande and Lambada. section 4.5
  • Varying the number of removed blocks and tracking benchmark retention to identify degradation. section 4.5

How Rearchitecting LLMs covers it

How deep it goes
Implements adjacency as a filter during importance-ranked selection, but does not establish that dispersed removals improve benchmarks.
Examples
Compares WinoGrande selections [25, 26, 24, 23] and [25, 23, 13, 15], with both protections active in the latter.
What it assumes
Assumes data-driven block-importance scores and a depth-pruning workflow.
What you can do after
Evaluate protection variants by pruning models and comparing WinoGrande and Lambada retention.
How it connects
Pairs block-importance ranking with depth pruning and heuristic protection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.