Static pruning reduces model size by selecting blocks or neurons from structural rules or parameter statistics, without analyzing behavior on real data. Selection is simple and incurs no computational cost, but can remove components that matter for a particular task. section 4.2 section 4.2.3
Common selection strategies include:
- Remove blocks by position, such as the first, middle, or final blocks; protection heuristics may preserve final blocks. section 4.2.3
- Rank blocks or neurons by weight magnitude and prune those selected by the criterion. section 4.3.5 section 5.2.5
Figure 4.3 Comparison of block selection strategies for depth pruning. From left to right: The full base model. Static strategies that remove blocks based on their position: the first, the last, or the middle ones. A data-driven strategy, where the removed blocks are scattered according to their actual contribution to the task. The data-driven strategy combined with a protection heuristic, which preserves the initial and final blocks.
Compare positional block removal and protection heuristics with data-driven selection. section 4.2.3
Benchmark trade-offs
In the reported comparisons, removing middle blocks retained 96.1% of baseline WinoGrande performance but 63.7% on Lambada; magnitude-selected blocks retained 88.6% and 3.2%, respectively. Static choices can therefore behave very differently across evaluations. section 4.3.5
