liveBook · Manning

Protection heuristic

As explained in Rearchitecting LLMs

A protection heuristic is a rule that excludes selected model components from pruning to reduce the risk of damaging model capabilities.

A protection heuristic excludes selected model components from pruning to reduce the risk of damaging capabilities. In depth pruning, it can preserve the first and last transformer blocks even when an importance score ranks them as removal candidates. Scores are guides, not guarantees: a low-scoring block may perform subtle transformations that matter for particular inputs. section 4.3.4

One selection workflow sorts blocks by increasing importance, skips protected indices, and collects candidates until it reaches the requested count. For the 28-block Qwen3-0.6B example, the fixed protected set is [0, 1, 2, 3, 26, 27]. A separate optional check skips a candidate if it is adjacent to an already selected block. Avoiding adjacent removals has a weaker justification than protecting the model’s endpoints, so it should be tested rather than treated as a fixed rule. section 4.3.4

Listing 4.8 Select blocks to remove.
def select_layers_to_prune(importance_scores,
                           num_layers=2,
                           heuristic_protection=True,
                           adjacent_protection=True):
 
    sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])
 
    selected = []
    for layer, score in sorted_layers:
        if heuristic_protection and layer in [0, 1, 2, 3, 26, 27]:  
            continue
 
        if adjacent_protection and any(abs(layer - l) == 1 for l in selected):
            continue
 
        selected.append(layer)
 
        if len(selected) >= num_layers:
            break
 
    return selected

The selector implements endpoint and adjacency protection as independent switches. Check the returned count: if too few candidates pass the safeguards, the function can return fewer blocks than requested. section 4.3.4

Calibration data and the importance metric affect which blocks are candidates. Static weight magnitude and WinoGrande-calibrated cosine similarity give different rankings, particularly near the end of the model. Fixed exclusions therefore add a safeguard that is independent of the ranking; they also restrict the available choices. section 4.3.4 section 4.4

Compare protected and unprotected pruning

For four removals guided by WinoGrande cosine importance, the unprotected selection is [25, 26, 24, 23]; protecting endpoints and avoiding adjacency produces [25, 23, 13, 15]. Against the unpruned baseline, the protected model retains 97.2% of WinoGrande performance and 73.1% of Lambada performance, versus 96.4% and 10.2% for the unprotected model. Both protections changed together, so this comparison does not isolate either one’s contribution. section 4.3.5

Table 4.3 Benchmark results
Model
Layers removed
WinoGrande
WinoGrande retention
Lambada
Lambada retention
Base
0.566
100%
0.404
100%
Mid. blocks.
[12, 13, 14, 15]
0.544
96.1%
0.257
63.7%
Mid. blocks adjacent protection
[11, 13, 15, 17]
0.502
88.7%
0.140
34.6%
Last blocks
[24, 25, 26, 27]
0.522
92.2%
0.0363
9%
Last blocks protected
[23, 24, 25, 26]
0.545
96.4%
0.0413
10.2%
Last blocks protected 2
[22, 23, 24, 25]
0.548
96.8%
0.224
55.5%
Magnitude
[8,2,7,6]
0.501
88.6%
0.0128
3.2%
Data guided
[25, 26, 24, 23]
0.545
96.4%
0.0413
10.2%
Data guided protected
[25, 23, 13, 15]
0.550
97.2%
0.2954
73.1%

Compare the baseline and pruning variants across both benchmarks; avoiding adjacent middle-block removals did not improve results in this comparison. section 4.3.5

To isolate the safeguards, disable endpoint protection and adjacency protection one at a time, then evaluate each model on WinoGrande and Lambada. Also vary the number of removed blocks to see where retention degrades. WinoGrande’s binary-choice format has a chance floor near 50%, so it can obscure damage that another benchmark reveals. section 4.5 section 4.3.5

How Rearchitecting LLMs covers it

How deep it goes
Implements endpoint and adjacency checks in a ranked selector, while treating adjacency protection as an option requiring evaluation.
Examples
Compares protected and unprotected Qwen3-0.6B selections on WinoGrande and Lambada, and proposes separate safeguard ablations.
What you can do after
Enables readers to evaluate protection heuristics by toggling safeguards and measuring retention across pruning configurations.
How it connects
Connects protection heuristics to depth pruning, transformer blocks, importance scoring, cosine similarity, and calibration datasets.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.