Domain specialization adapts a general model to a particular domain or task. It can reduce the cost and lack of differentiation associated with oversized generic models, but pruning reduces model capacity and can worsen general performance. section 1.5 section 5.2.5
For data-driven width pruning, collect activation norms from domain-specific calibration data. Combine these dynamic activity scores with static weight-magnitude scores to rank neuron pairs, prune the selected pairs, and replace the MLP layers. Check that activation data exists for every layer and update the model configuration to avoid library compatibility issues. section 5.2.4
Listing 5.7 Update model for data-driven pruning
def update_model(model, prune_percent, activation_norms):
new_intermediate_size = None
pruning_stats = []
for idx, layer in enumerate(model.model.layers): #A
mlp = layer.mlp
if idx not in activation_norms: #1
raise KeyError(f"No activation norms for layer {idx}")
X_d_norm = activation_norms[idx] #2
original_size = mlp.gate_proj.out_features
new_gate_proj, new_up_proj, new_down_proj, new_size = prune_neuron_pairs(
mlp=mlp,
prune_percent=prune_percent,
X_d_norm=X_d_norm, #3
layer_idx=idx
)
mlp.gate_proj = new_gate_proj
mlp.up_proj = new_up_proj
mlp.down_proj = new_down_proj
if new_intermediate_size is None:
new_intermediate_size = new_size
model.config.intermediate_size = new_intermediate_size #4
return model
Shows the layer-by-layer update function, including its activation-data check and the pruning call. section 5.2.4
Wiki and SMS variants
Calibration on wiki text and SMS data produces different pruned Llama-3.2-1B variants, despite the same pruning percentage and parameter count. Each variant has lower perplexity on its own calibration domain than the other specialized variant, but both score worse than the base model. Treat pruning as damage minimization before recovery, not as the complete pipeline. section 5.2.4 section 5.2.5
Table 5.1 Perplexity results
|
Model
|
Parameters
|
Wiki
Perplexity
|
SMS
Perplexity
|
|
Base
|
1.24B
|
25.69
|
122.91
|
|
Wiki model
|
1.07B
(~87%)
|
36.28
|
182.54
|
|
SMS model
|
1.07B
(~87%)
|
48.65
|
165.54
|
|
Static 20% pruned
|
1.07B
(~87%)
|
52
|
217
|
Compares base, domain-pruned, and static-pruned models by parameter count and wiki and SMS perplexity. section 5.2.5



