In a GLU-based MLP, each intermediate neuron corresponds to a row in both gate_proj and up_proj. Width pruning removes paired neurons while retaining hidden_size at MLP boundaries. section 5.1.1 section 5.5
Neuron selection can use:
- Static scoring: combine the weight ranges for each row in gate_proj and up_proj; lower scores mark removal candidates. section 5.1.1
- Data-driven scoring: combine structural importance with activation norms measured during calibration. section 5.5
def compute_neuron_pair_importance(gate_weight, up_weight):
gate_max_abs = torch.max(gate_weight, dim=1).values + ➥
torch.abs(torch.min(gate_weight, dim=1).values)
up_max_abs = torch.max(up_weight, dim=1).values + ➥
torch.abs(torch.min(up_weight, dim=1).values)
importance_scores = gate_max_abs + up_max_abs
return importance_scores
Computes a score for each paired neuron by summing the gate and up projection weight ranges. section 5.1.1
Comparing weight ranges
A neuron with weights from -0.2 to 0.1 has a range score of 0.3; one spanning -0.5 to 0.4 scores 0.9 and is less likely to be removed. section 5.1.1









