liveBook · Manning

Structured pruning

As explained in Rearchitecting LLMs

Structured pruning is a model-compression technique that removes organized groups of parameters, such as neurons, channels, or layers.

Structured pruning compresses a model by physically removing organized parts of its architecture. Depth pruning removes entire Transformer layers to reduce model size and speed up inference; width pruning changes another architectural dimension. Both can cause knowledge loss, which generally increases with the amount pruned. section 1.0 section 2.8 section 6.0

Choose what to remove against the target task, not just a desired size. Pruning by block importance recovered performance in less training time than removing the same number of final blocks in one comparison. For a narrow task, specialization can make more aggressive pruning practical than when general capabilities must be retained. section 6.0 section 12.1

A depth-pruning workflow starts with a baseline and checks the cost of each intervention:

  • Measure the original model on relevant benchmarks before changing its architecture. section 2.8
  • Select layers to remove. Compare the default later-layer strategy with first-layer or specific non-adjacent selections. section 2.7
  • Evaluate the pruned model, then recover capability through knowledge distillation from a larger teacher. Compare recovery and benchmark results against the baseline. section 2.7 section 2.8
  • Vary the number of removed layers and the recovery dataset size. Check whether task-aligned recovery data improves the target benchmark at the expense of others. section 2.7
student_model = prune_model(
    model=copy.deepcopy(teacher_model),
    pruning_type="DEPTH",
    layer_indices=[2, 5, 8],
    show_progress=True
)

This depth-pruning call copies the teacher and removes the specified non-adjacent layers to create a student model. section 2.7

Domain-guided pruning

In the FineScope example, domain-specific data guides which components to preserve in Llama 3.1 8B. Accuracy remains stable up to around 35% pruning, while pruning guided by generic data begins to degrade at 25%. section 1.1.2

Recovery depends on both the parts retained and the training method. Distillation can use teacher outputs, dataset labels, and intermediate-block alignment; aligning blocks becomes more complicated when teacher and student have different numbers of layers. section 6.0

How Rearchitecting LLMs covers it

How deep it goes
Provides a depth-pruning workflow and layer-selection experiments, while identifying width pruning without detailing its implementation.
Examples
Uses non-adjacent layer removal and a domain-guided Llama 3.1 8B pruning comparison.
What you can do after
Practice pruning selected layers and comparing capability recovery across layer choices and distillation datasets.
How it connects
Connects pruning to knowledge distillation, model specialization, benchmark evaluation, and Transformer layers.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.