Adjacent block protection prevents a depth-pruning selector from removing consecutive transformer blocks. Applied during importance-ranked selection, it skips a candidate if a neighboring block has already been selected. It can be enabled independently of heuristic protection for early and late blocks. section 4.3.4
WinoGrande block selection
For four removals, the unprotected selection is [25, 26, 24, 23]. With adjacent and heuristic protection both enabled, it is [25, 23, 13, 15]. The comparison changes the selection strategy, not the number of blocks removed; benchmark evaluation is needed to assess the effect. section 4.3.4
Listing 4.8 Select blocks to remove.
def select_layers_to_prune(importance_scores,
num_layers=2,
heuristic_protection=True,
adjacent_protection=True):
sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])
selected = []
for layer, score in sorted_layers:
if heuristic_protection and layer in [0, 1, 2, 3, 26, 27]:
continue
if adjacent_protection and any(abs(layer - l) == 1 for l in selected):
continue
selected.append(layer)
if len(selected) >= num_layers:
break
return selected
The selector shows how adjacency protection skips a candidate next to a previously selected block. section 4.3.4
Test protection choices by:
- Disabling heuristic and adjacent protection separately, then evaluating each resulting model on WinoGrande and Lambada. section 4.5
- Varying the number of removed blocks and tracking benchmark retention to identify degradation. section 4.5
