liveBook · Manning

Long-range context

As explained in Rearchitecting LLMs

Long-range context is information occurring substantially earlier in a sequence that remains relevant to predicting a later token.

Long-range context is information earlier in a sequence that remains relevant to a later prediction. Preserving it matters when pruning a model: removing layers can reduce inference cost or memory use while weakening the model’s ability to integrate information across a passage. section 4.3.5 section 8.4.2

LAMBADA tests this capability by asking a model to predict the exact final word of a passage. The answer requires the whole paragraph, not just the last sentence. Unlike a closed-set task, it offers no distractors to eliminate; a model that loses the relevant capability can fall close to zero. section 4.3.5 section 8.4.2

Depth pruning and LAMBADA retention

Relative to a base LAMBADA score of 0.404, removing four consecutive middle blocks retained 63.7% of the score. A protected data-guided selection retained 73.1%, while removing the last four blocks retained 9%. Spreading removals across nonconsecutive middle positions retained 34.6%, less than removing a consecutive middle chunk. section 4.3.5

Table 4.3 Benchmark results
Model
Layers removed
WinoGrande
WinoGrande retention
Lambada
Lambada retention
Base
0.566
100%
0.404
100%
Mid. blocks.
[12, 13, 14, 15]
0.544
96.1%
0.257
63.7%
Mid. blocks adjacent protection
[11, 13, 15, 17]
0.502
88.7%
0.140
34.6%
Last blocks
[24, 25, 26, 27]
0.522
92.2%
0.0363
9%
Last blocks protected
[23, 24, 25, 26]
0.545
96.4%
0.0413
10.2%
Last blocks protected 2
[22, 23, 24, 25]
0.548
96.8%
0.224
55.5%
Magnitude
[8,2,7,6]
0.501
88.6%
0.0128
3.2%
Data guided
[25, 26, 24, 23]
0.545
96.4%
0.0413
10.2%
Data guided protected
[25, 23, 13, 15]
0.550
97.2%
0.2954
73.1%

Compare LAMBADA and WinoGrande scores across the same depth-pruning variants; their different task formats expose different degrees of degradation. section 4.3.5

Attention pruning makes a related quality–memory trade-off. Removing attention layers reduces KV-cache entries per generated token, lowering dynamic VRAM growth. In the evaluated 3B model, LAMBADA was progressively sensitive across pruning ratios, so memory savings alone do not establish that context-dependent generation remains reliable. section 8.4.2

Table 8.2 Dynamic VRAM delta across pruning configurations
Configuration
Dynamic VRAM
Vs Baseline
Baseline
109.57 MB
_
3 layers removed
96.06 MB
-12.3%
6 layers removed
78.72 MB
-28.2%
15 layers removed
58.63 MB
-46.5%

Compare dynamic VRAM growth during 500-token generation against the loaded-model baseline for each attention-pruning configuration. section 8.4.2

How Rearchitecting LLMs covers it

How deep it goes
Explains the prediction requirement and compares pruning outcomes, without specifying an implementation for contextual integration.
Examples
Compares LAMBADA retention across middle-block, final-block, and protected data-guided removals, and reports attention-pruning memory measurements.
What you can do after
Evaluate pruning variants for context-dependent prediction alongside inference and memory savings.
How it connects
Connects long-range context to LAMBADA, depth pruning, attention pruning, KV cache, and dynamic VRAM.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.