liveBook · Manning

Cosine importance

As explained in Rearchitecting LLMs

Cosine importance estimates a model component's importance using cosine similarity between related activation or representation vectors.

Cosine importance estimates block importance by comparing related activations or representations. In depth pruning, the ranking helps select blocks to remove; calibration data and task can affect the selection. section 4.3.5 section 4.6

Data-guided block selection

Winogrande cosine importance selected blocks [25, 26, 24, 23] as the four least important. Protecting the last two blocks and avoiding consecutive removals instead selected [25, 23, 13, 15]. section 4.3.5

Table 4.3 Benchmark results
Model
Layers removed
WinoGrande
WinoGrande retention
Lambada
Lambada retention
Base
0.566
100%
0.404
100%
Mid. blocks.
[12, 13, 14, 15]
0.544
96.1%
0.257
63.7%
Mid. blocks adjacent protection
[11, 13, 15, 17]
0.502
88.7%
0.140
34.6%
Last blocks
[24, 25, 26, 27]
0.522
92.2%
0.0363
9%
Last blocks protected
[23, 24, 25, 26]
0.545
96.4%
0.0413
10.2%
Last blocks protected 2
[22, 23, 24, 25]
0.548
96.8%
0.224
55.5%
Magnitude
[8,2,7,6]
0.501
88.6%
0.0128
3.2%
Data guided
[25, 26, 24, 23]
0.545
96.4%
0.0413
10.2%
Data guided protected
[25, 23, 13, 15]
0.550
97.2%
0.2954
73.1%

Compare benchmark scores and retention for cosine-guided selections and other depth-pruning strategies. section 4.3.5

How Rearchitecting LLMs covers it

How deep it goes
Uses cosine-based rankings to compare block removals on task benchmarks.
Examples
Selects four blocks using Winogrande cosine importance, then tests a protected selection variant.
What it assumes
Requires calibration data for ranking; recognizes that data and task can change block importance.
What you can do after
Apply it to rank blocks for data-guided depth pruning and compare protected removal choices.
How it connects
Connects depth pruning with calibration data, model activations, and task performance.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.