The expansion ratio, or expansion rate, sets an MLP’s intermediate width relative to the model’s hidden-state width: intermediate size ÷ hidden-state size. It can be expressed as a factor or percentage. In width pruning, specifying the target ratio sets the final intermediate size directly, rather than specifying a pruning percentage. section 13.4.1
A lower ratio can reduce model size and computational cost, but the effect on task performance depends on the model. Experiments with Llama-3.2-1B and 3B found that some lower ratios could improve performance on certain tasks; experiments with the Gemma family gave different results. section 3.2.2
From 300% to 225%
With a hidden-state size of 1024, an intermediate size of 3072 gives an expansion rate of 300%. A target of 225% gives 2304 neurons: a 25% reduction in intermediate size, while the hidden-state size stays fixed. section 13.4.1
For a GLU MLP, gate_proj and up_proj work in pairs. Pruning must evaluate neuron pairs jointly and remove corresponding neurons from both projections so their element-wise multiplication remains valid. section 3.2.2
Listing 13.13 Width pruning
EXPANSION_RATE = 225
EXPANSION_DIVISOR = 128
model_m_pruned, width_stats = opf.prune_model(
model=copy.deepcopy(depth_pruned_model),
pruning_type="MLP_GLU",
neuron_selection_method="PPM", #1
pruning_percentage=None, #A
expansion_rate=EXPANSION_RATE, #2
expansion_divisor=EXPANSION_DIVISOR, #3
dataloader=calibration_dataloader, #B
show_progress=True,
return_stats=True,
)
model_m_pruned.generation_config.pad_token_id = tokenizer.pad_token_id
model_m_pruned.eval()
inter = [layer.mlp.gate_proj.out_features for layer in model_m_pruned.model.layers]
print(f"layers: {len(inter)}")
print(f"intermediate sizes: {Counter(inter)}")This width-pruning workflow sets expansion_rate to 225 and expansion_divisor to 128, then checks gate_proj output widths across the pruned model. The divisor makes the intermediate size a multiple of 128, which usually makes better use of the GPU. section 13.4.1
Validate both architecture and task performance after pruning. In one specialist cascade, all 25 blocks reached an intermediate size of 2304. After depth and width pruning, valid JSON fell from 100% to 74.1%, and exact match fell from 96.3% to 51.9% against the original specialist. section 13.4.1
