Structural optimization changes a model’s architecture to reduce computational requirements or adapt it to a task. Pruning removes components that contribute least to the objective, but the impact on performance depends on the technique and extent of restructuring. section 1.2 section 1.2.1
Pruning can operate at different scales:
- Remove entire blocks for aggressive restructuring; this may require knowledge recovery. section 1.2.1
- Remove a small number of neurons for more targeted changes that can preserve general capabilities. section 1.2.1
A common pipeline prunes the model, recovers lost capabilities through knowledge distillation, and optionally specializes it through fine-tuning. Some projects use only pruning and recovery. section 1.2 section 1.5
Figure 1.1 The model tailoring pipeline consists of core phases (shown with solid arrows) and optional phases (shown with dashed arrows). In the first phase, we adapt the structure to the model's objectives through pruning. Next, we recover capabilities it may have lost through knowledge distillation. Finally, we can optionally specialize the model through fine-tuning. An optional initial phase calibrates the base model, via a brief fine-tuning pass on the target dataset.
See how pruning, knowledge recovery, and optional specialization fit into the model-tailoring pipeline. section 1.2
student_model = prune_model(
model=copy.deepcopy(teacher_model),
pruning_type="DEPTH",
layer_indices=[2, 5, 8],
show_progress=True
)
Example: apply depth pruning to layers 2, 5, and 8. section 2.7
