Long-range context is information earlier in a sequence that remains relevant to a later prediction. Preserving it matters when pruning a model: removing layers can reduce inference cost or memory use while weakening the model’s ability to integrate information across a passage. section 4.3.5 section 8.4.2
LAMBADA tests this capability by asking a model to predict the exact final word of a passage. The answer requires the whole paragraph, not just the last sentence. Unlike a closed-set task, it offers no distractors to eliminate; a model that loses the relevant capability can fall close to zero. section 4.3.5 section 8.4.2
Depth pruning and LAMBADA retention
Relative to a base LAMBADA score of 0.404, removing four consecutive middle blocks retained 63.7% of the score. A protected data-guided selection retained 73.1%, while removing the last four blocks retained 9%. Spreading removals across nonconsecutive middle positions retained 34.6%, less than removing a consecutive middle chunk. section 4.3.5
Table 4.3 Benchmark results
|
Model
Layers removed
|
WinoGrande
|
WinoGrande retention
|
Lambada
|
Lambada retention
|
|
Base
|
0.566
|
100%
|
0.404
|
100%
|
|
Mid. blocks.
[12, 13, 14, 15]
|
0.544
|
96.1%
|
0.257
|
63.7%
|
|
Mid. blocks adjacent protection
[11, 13, 15, 17]
|
0.502
|
88.7%
|
0.140
|
34.6%
|
|
Last blocks
[24, 25, 26, 27]
|
0.522
|
92.2%
|
0.0363
|
9%
|
|
Last blocks protected
[23, 24, 25, 26]
|
0.545
|
96.4%
|
0.0413
|
10.2%
|
|
Last blocks protected 2
[22, 23, 24, 25]
|
0.548
|
96.8%
|
0.224
|
55.5%
|
|
Magnitude
[8,2,7,6]
|
0.501
|
88.6%
|
0.0128
|
3.2%
|
|
Data guided
[25, 26, 24, 23]
|
0.545
|
96.4%
|
0.0413
|
10.2%
|
|
Data guided protected
[25, 23, 13, 15]
|
0.550
|
97.2%
|
0.2954
|
73.1%
|
Compare LAMBADA and WinoGrande scores across the same depth-pruning variants; their different task formats expose different degrees of degradation. section 4.3.5
Attention pruning makes a related quality–memory trade-off. Removing attention layers reduces KV-cache entries per generated token, lowering dynamic VRAM growth. In the evaluated 3B model, LAMBADA was progressively sensitive across pruning ratios, so memory savings alone do not establish that context-dependent generation remains reliable. section 8.4.2
Table 8.2 Dynamic VRAM delta across pruning configurations
|
Configuration
|
Dynamic VRAM
|
Vs Baseline
|
|
Baseline
|
109.57 MB
|
_
|
|
3 layers removed
|
96.06 MB
|
-12.3%
|
|
6 layers removed
|
78.72 MB
|
-28.2%
|
|
15 layers removed
|
58.63 MB
|
-46.5%
|
Compare dynamic VRAM growth during 500-token generation against the loaded-model baseline for each attention-pruning configuration. section 8.4.2
