liveBook · Manning

Static pruning

As explained in Rearchitecting LLMs

Static pruning selects model components for removal using fixed structural information or parameter statistics rather than runtime activation behavior.

Static pruning reduces model size by selecting blocks or neurons from structural rules or parameter statistics, without analyzing behavior on real data. Selection is simple and incurs no computational cost, but can remove components that matter for a particular task. section 4.2 section 4.2.3

Common selection strategies include:

  • Remove blocks by position, such as the first, middle, or final blocks; protection heuristics may preserve final blocks. section 4.2.3
  • Rank blocks or neurons by weight magnitude and prune those selected by the criterion. section 4.3.5 section 5.2.5
Figure 4.3 Comparison of block selection strategies for depth pruning. From left to right: The full base model. Static strategies that remove blocks based on their position: the first, the last, or the middle ones. A data-driven strategy, where the removed blocks are scattered according to their actual contribution to the task. The data-driven strategy combined with a protection heuristic, which preserves the initial and final blocks.

Compare positional block removal and protection heuristics with data-driven selection. section 4.2.3

Benchmark trade-offs

In the reported comparisons, removing middle blocks retained 96.1% of baseline WinoGrande performance but 63.7% on Lambada; magnitude-selected blocks retained 88.6% and 3.2%, respectively. Static choices can therefore behave very differently across evaluations. section 4.3.5

How Rearchitecting LLMs covers it

How deep it goes
Defines structural and magnitude-based selection, compares positional heuristics, and identifies risks without measuring task-specific behavior.
Examples
Compares positional and magnitude-based block removal, plus magnitude-pruned neurons evaluated with perplexity.
What you can do after
Evaluate candidate removals against benchmarks while recognizing that static heuristics may discard task-critical capabilities.
How it connects
Contrasts with data-driven pruning and relates to depth pruning, width pruning, and knowledge distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.