Structured pruning compresses a model by physically removing organized parts of its architecture. Depth pruning removes entire Transformer layers to reduce model size and speed up inference; width pruning changes another architectural dimension. Both can cause knowledge loss, which generally increases with the amount pruned. section 1.0 section 2.8 section 6.0
Choose what to remove against the target task, not just a desired size. Pruning by block importance recovered performance in less training time than removing the same number of final blocks in one comparison. For a narrow task, specialization can make more aggressive pruning practical than when general capabilities must be retained. section 6.0 section 12.1
A depth-pruning workflow starts with a baseline and checks the cost of each intervention:
- Measure the original model on relevant benchmarks before changing its architecture. section 2.8
- Select layers to remove. Compare the default later-layer strategy with first-layer or specific non-adjacent selections. section 2.7
- Evaluate the pruned model, then recover capability through knowledge distillation from a larger teacher. Compare recovery and benchmark results against the baseline. section 2.7 section 2.8
- Vary the number of removed layers and the recovery dataset size. Check whether task-aligned recovery data improves the target benchmark at the expense of others. section 2.7
student_model = prune_model(
model=copy.deepcopy(teacher_model),
pruning_type="DEPTH",
layer_indices=[2, 5, 8],
show_progress=True
)
This depth-pruning call copies the teacher and removes the specified non-adjacent layers to create a student model. section 2.7
Domain-guided pruning
In the FineScope example, domain-specific data guides which components to preserve in Llama 3.1 8B. Accuracy remains stable up to around 35% pruning, while pruning guided by generic data begins to degrade at 25%. section 1.1.2
Recovery depends on both the parts retained and the training method. Distillation can use teacher outputs, dataset labels, and intermediate-block alignment; aligning blocks becomes more complicated when teacher and student have different numbers of layers. section 6.0
