A protection heuristic excludes selected model components from pruning to reduce the risk of damaging capabilities. In depth pruning, it can preserve the first and last transformer blocks even when an importance score ranks them as removal candidates. Scores are guides, not guarantees: a low-scoring block may perform subtle transformations that matter for particular inputs. section 4.3.4
One selection workflow sorts blocks by increasing importance, skips protected indices, and collects candidates until it reaches the requested count. For the 28-block Qwen3-0.6B example, the fixed protected set is [0, 1, 2, 3, 26, 27]. A separate optional check skips a candidate if it is adjacent to an already selected block. Avoiding adjacent removals has a weaker justification than protecting the model’s endpoints, so it should be tested rather than treated as a fixed rule. section 4.3.4
Listing 4.8 Select blocks to remove.
def select_layers_to_prune(importance_scores,
num_layers=2,
heuristic_protection=True,
adjacent_protection=True):
sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])
selected = []
for layer, score in sorted_layers:
if heuristic_protection and layer in [0, 1, 2, 3, 26, 27]:
continue
if adjacent_protection and any(abs(layer - l) == 1 for l in selected):
continue
selected.append(layer)
if len(selected) >= num_layers:
break
return selected
The selector implements endpoint and adjacency protection as independent switches. Check the returned count: if too few candidates pass the safeguards, the function can return fewer blocks than requested. section 4.3.4
Calibration data and the importance metric affect which blocks are candidates. Static weight magnitude and WinoGrande-calibrated cosine similarity give different rankings, particularly near the end of the model. Fixed exclusions therefore add a safeguard that is independent of the ranking; they also restrict the available choices. section 4.3.4 section 4.4
Compare protected and unprotected pruning
For four removals guided by WinoGrande cosine importance, the unprotected selection is [25, 26, 24, 23]; protecting endpoints and avoiding adjacency produces [25, 23, 13, 15]. Against the unpruned baseline, the protected model retains 97.2% of WinoGrande performance and 73.1% of Lambada performance, versus 96.4% and 10.2% for the unprotected model. Both protections changed together, so this comparison does not isolate either one’s contribution. section 4.3.5
Table 4.3 Benchmark results
|
Model
Layers removed
|
WinoGrande
|
WinoGrande retention
|
Lambada
|
Lambada retention
|
|
Base
|
0.566
|
100%
|
0.404
|
100%
|
|
Mid. blocks.
[12, 13, 14, 15]
|
0.544
|
96.1%
|
0.257
|
63.7%
|
|
Mid. blocks adjacent protection
[11, 13, 15, 17]
|
0.502
|
88.7%
|
0.140
|
34.6%
|
|
Last blocks
[24, 25, 26, 27]
|
0.522
|
92.2%
|
0.0363
|
9%
|
|
Last blocks protected
[23, 24, 25, 26]
|
0.545
|
96.4%
|
0.0413
|
10.2%
|
|
Last blocks protected 2
[22, 23, 24, 25]
|
0.548
|
96.8%
|
0.224
|
55.5%
|
|
Magnitude
[8,2,7,6]
|
0.501
|
88.6%
|
0.0128
|
3.2%
|
|
Data guided
[25, 26, 24, 23]
|
0.545
|
96.4%
|
0.0413
|
10.2%
|
|
Data guided protected
[25, 23, 13, 15]
|
0.550
|
97.2%
|
0.2954
|
73.1%
|
Compare the baseline and pruning variants across both benchmarks; avoiding adjacent middle-block removals did not improve results in this comparison. section 4.3.5
To isolate the safeguards, disable endpoint protection and adjacency protection one at a time, then evaluate each model on WinoGrande and Lambada. Also vary the number of removed blocks to see where retention degrades. WinoGrande’s binary-choice format has a chance floor near 50%, so it can obscure damage that another benchmark reveals. section 4.5 section 4.3.5
