Cosine importance estimates block importance by comparing related activations or representations. In depth pruning, the ranking helps select blocks to remove; calibration data and task can affect the selection. section 4.3.5 section 4.6
Data-guided block selection
Winogrande cosine importance selected blocks [25, 26, 24, 23] as the four least important. Protecting the last two blocks and avoiding consecutive removals instead selected [25, 23, 13, 15]. section 4.3.5
Table 4.3 Benchmark results
|
Model
Layers removed
|
WinoGrande
|
WinoGrande retention
|
Lambada
|
Lambada retention
|
|
Base
|
0.566
|
100%
|
0.404
|
100%
|
|
Mid. blocks.
[12, 13, 14, 15]
|
0.544
|
96.1%
|
0.257
|
63.7%
|
|
Mid. blocks adjacent protection
[11, 13, 15, 17]
|
0.502
|
88.7%
|
0.140
|
34.6%
|
|
Last blocks
[24, 25, 26, 27]
|
0.522
|
92.2%
|
0.0363
|
9%
|
|
Last blocks protected
[23, 24, 25, 26]
|
0.545
|
96.4%
|
0.0413
|
10.2%
|
|
Last blocks protected 2
[22, 23, 24, 25]
|
0.548
|
96.8%
|
0.224
|
55.5%
|
|
Magnitude
[8,2,7,6]
|
0.501
|
88.6%
|
0.0128
|
3.2%
|
|
Data guided
[25, 26, 24, 23]
|
0.545
|
96.4%
|
0.0413
|
10.2%
|
|
Data guided protected
[25, 23, 13, 15]
|
0.550
|
97.2%
|
0.2954
|
73.1%
|
Compare benchmark scores and retention for cosine-guided selections and other depth-pruning strategies. section 4.3.5
