liveBook · Manning

Peak-to-peak magnitude pruning

As explained in Rearchitecting LLMs

Peak-to-peak magnitude pruning ranks neurons using the peak-to-peak magnitude of their activations or weights.

Peak-to-peak magnitude (PPM) ranks neurons for MLP width pruning. In hybrid mode, it combines neuron weights with activation statistics measured on calibration prompts. section 13.4.1

With OptiPFair, select PPM for MLP-GLU modules and set the target width using an expansion rate. A divisor can constrain the resulting intermediate size to a multiple suited to GPU use. section 13.4.1

Listing 13.13 Width pruning
EXPANSION_RATE = 225              
EXPANSION_DIVISOR = 128        
   
model_m_pruned, width_stats = opf.prune_model(
    model=copy.deepcopy(depth_pruned_model),
    pruning_type="MLP_GLU",
    neuron_selection_method="PPM", #1    
    pruning_percentage=None, #A           
    expansion_rate=EXPANSION_RATE, #2
    expansion_divisor=EXPANSION_DIVISOR, #3
    dataloader=calibration_dataloader, #B
    show_progress=True,
    return_stats=True,
)
model_m_pruned.generation_config.pad_token_id = tokenizer.pad_token_id
model_m_pruned.eval()
 
inter = [layer.mlp.gate_proj.out_features for layer in model_m_pruned.model.layers]
print(f"layers: {len(inter)}")
print(f"intermediate sizes: {Counter(inter)}")

See the OptiPFair call that applies PPM to a calibrated model and sets its target expansion rate and divisor. section 13.4.1

How Rearchitecting LLMs covers it

How deep it goes
Combines neuron weights with activation statistics from calibration prompts, without detailing the ranking calculation.
Examples
Uses a 225% expansion rate to reduce 3072 intermediate dimensions to 2304 across 25 layers.
What it assumes
Prepare a calibration dataloader and an MLP-GLU model for width pruning.
What you can do after
Apply it via OptiPFair to prune MLP-GLU width using calibration data and a target expansion rate.
How it connects
Connects neuron ranking to activation statistics, calibration data, and MLP-GLU width pruning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.